2024-05-26T20:05:12.154 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-26T20:05:12.154 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi --verbose --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} --archive /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509 -- /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/orig.config.yaml 2024-05-26T20:05:12.188 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-26T20:05:12.266 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509 branch: reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} email: ceph-qa@ceph.com first_in_suite: false job_id: '7725509' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 1296 sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBCfhtYy3Qw3mwMAX9p72Gz0S7a+LiV/khZYrpw0xSead6z7IlHJ4uA9wVBZvncokQwdr9xZoMKI5ZkZ3+i33c= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFdhq4LVhtbDhGEnFztzQHEDXgbHB7CDfQGa/mAGD0p7mZ1B2lRDq+A7AtI9SE0/anQeadF/vf9q0uiaPCFTsQ= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-24_22:40:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-26T20:05:12.266 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa; will attempt to use it 2024-05-26T20:05:12.267 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks 2024-05-26T20:05:12.267 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-26T20:05:12.268 INFO:teuthology.task.internal:Checking packages... 2024-05-26T20:05:12.287 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc' 2024-05-26T20:05:12.287 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-26T20:05:12.287 INFO:teuthology.packaging:ref: None 2024-05-26T20:05:12.288 INFO:teuthology.packaging:tag: None 2024-05-26T20:05:12.288 INFO:teuthology.packaging:branch: reef 2024-05-26T20:05:12.288 INFO:teuthology.packaging:sha1: f1882d8093fb4b60c1d1e52444565787ca4bd6cc 2024-05-26T20:05:12.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-05-26T20:05:12.586 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1363.g19f3c031 2024-05-26T20:05:12.587 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-26T20:05:12.594 INFO:teuthology.task.internal:no buildpackages task found 2024-05-26T20:05:12.594 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-26T20:05:12.621 INFO:teuthology.task.internal:Saving configuration 2024-05-26T20:05:12.634 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-26T20:05:12.647 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-26T20:05:12.671 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 20:00:24.456721', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBCfhtYy3Qw3mwMAX9p72Gz0S7a+LiV/khZYrpw0xSead6z7IlHJ4uA9wVBZvncokQwdr9xZoMKI5ZkZ3+i33c='} 2024-05-26T20:05:12.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 20:00:24.458230', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFdhq4LVhtbDhGEnFztzQHEDXgbHB7CDfQGa/mAGD0p7mZ1B2lRDq+A7AtI9SE0/anQeadF/vf9q0uiaPCFTsQ='} 2024-05-26T20:05:12.697 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-26T20:05:12.703 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-05-26T20:05:12.703 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-05-26T20:05:12.703 INFO:teuthology.run_tasks:Running task console_log... 2024-05-26T20:05:12.783 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ce9c83be0>, signals=[15]) 2024-05-26T20:05:12.783 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-26T20:05:12.790 INFO:teuthology.task.internal:Opening connections... 2024-05-26T20:05:12.790 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:05:12.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:05:12.864 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:05:12.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:05:12.938 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-26T20:05:12.944 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-05-26T20:05:12.966 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-05-26T20:05:12.967 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-26T20:05:13.029 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-05-26T20:05:13.030 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-26T20:05:13.030 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-05-26T20:05:13.030 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-26T20:05:13.030 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-26T20:05:13.030 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-26T20:05:13.030 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-05-26T20:05:13.052 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-05-26T20:05:13.075 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-05-26T20:05:13.075 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-05-26T20:05:13.135 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-26T20:05:13.136 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-26T20:05:13.137 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-05-26T20:05:13.160 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-26T20:05:13.169 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-26T20:05:13.263 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-26T20:05:13.263 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-05-26T20:05:13.265 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-05-26T20:05:13.286 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-26T20:05:13.303 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-26T20:05:13.303 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-05-26T20:05:13.328 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-05-26T20:05:13.364 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-26T20:05:13.415 INFO:teuthology.run_tasks:Running task kernel... 2024-05-26T20:05:13.428 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-26T20:05:13.428 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-26T20:05:13.429 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-26T20:05:13.429 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-26T20:05:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T20:05:13.450 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-05-26T20:05:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T20:05:13.451 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-05-26T20:05:13.512 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-489.el8.x86_64 2024-05-26T20:05:13.512 INFO:teuthology.task.kernel:Running kernel on smithi007: 4.18.0-489.el8.x86_64 2024-05-26T20:05:13.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-05-26T20:05:13.513 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2024-05-26T20:05:13.513 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2024-05-26T20:05:13.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-05-26T20:05:17.189 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - AppStream 26 MB/s | 28 MB 00:01 2024-05-26T20:05:17.252 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 24 MB/s | 28 MB 00:01 2024-05-26T20:05:23.010 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 10 MB 00:00 2024-05-26T20:05:23.100 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 10 MB 00:00 2024-05-26T20:05:25.169 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-05-26T20:05:25.356 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2024-05-26T20:05:25.645 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 8.0 kB 00:00 2024-05-26T20:05:25.832 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 8.0 kB 00:00 2024-05-26T20:05:25.904 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-05-26T20:05:26.092 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-05-26T20:05:26.874 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 15 MB/s | 14 MB 00:00 2024-05-26T20:05:28.984 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 14 MB 00:02 2024-05-26T20:05:31.172 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 323 kB/s | 24 kB 00:00 2024-05-26T20:05:33.543 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 346 kB/s | 24 kB 00:00 2024-05-26T20:05:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-26T20:05:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-26T20:05:34.414 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-552.3.1.el8 baseos 10 M 2024-05-26T20:05:34.415 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-552.3.1.el8 baseos 44 M 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-552.3.1.el8 baseos 36 M 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:05:34.416 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 91 M 2024-05-26T20:05:34.417 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2024-05-26T20:05:34.417 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:05:35.033 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-552.3.1.el8.x86_64.rpm 25 MB/s | 10 MB 00:00 2024-05-26T20:05:35.567 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-modules-4.18.0-552.3.1.el8.x86_64 38 MB/s | 36 MB 00:00 2024-05-26T20:05:36.000 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-core-4.18.0-552.3.1.el8.x86_64.rp 32 MB/s | 44 MB 00:01 2024-05-26T20:05:36.001 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:05:36.001 INFO:teuthology.orchestra.run.smithi137.stdout:Total 57 MB/s | 91 MB 00:01 2024-05-26T20:05:36.692 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:05:36.798 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-26T20:05:36.798 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-26T20:05:36.919 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:05:36.919 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 4.18.0-552.3.1.el8 baseos 10 M 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 4.18.0-552.3.1.el8 baseos 44 M 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 4.18.0-552.3.1.el8 baseos 36 M 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:Install 3 Packages 2024-05-26T20:05:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:05:36.921 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 91 M 2024-05-26T20:05:36.921 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 96 M 2024-05-26T20:05:36.921 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:05:37.121 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:05:37.121 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:05:37.519 INFO:teuthology.orchestra.run.smithi007.stdout:(1/3): kernel-4.18.0-552.3.1.el8.x86_64.rpm 25 MB/s | 10 MB 00:00 2024-05-26T20:05:38.111 INFO:teuthology.orchestra.run.smithi007.stdout:(2/3): kernel-modules-4.18.0-552.3.1.el8.x86_64 36 MB/s | 36 MB 00:00 2024-05-26T20:05:38.379 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:05:38.400 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:05:38.611 INFO:teuthology.orchestra.run.smithi007.stdout:(3/3): kernel-core-4.18.0-552.3.1.el8.x86_64.rp 29 MB/s | 44 MB 00:01 2024-05-26T20:05:38.612 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:05:38.612 INFO:teuthology.orchestra.run.smithi007.stdout:Total 54 MB/s | 91 MB 00:01 2024-05-26T20:05:39.333 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:05:39.795 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:05:39.795 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:05:41.112 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:05:41.137 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:05:41.223 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:05:41.712 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:05:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:05:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:05:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:05:45.720 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:05:48.364 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:05:48.753 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:05:49.168 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:05:49.435 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:05:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:05:52.722 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:11.297 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:11.322 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:06:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:06:12.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:12.162 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:06:12.162 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:06:12.162 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.163 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.163 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.163 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:06:12.163 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:06:12.324 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T20:06:12.542 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.543 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.543 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.543 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-26T20:06:12.543 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-26T20:06:12.543 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-05-26T20:06:12.543 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-05-26T20:06:12.543 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T20:06:12.754 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.755 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-26T20:06:12.789 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:06:12.790 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-26T20:06:12.791 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-26T20:06:12.791 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-26T20:06:12.791 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-05-26T20:06:12.791 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:06:12.791 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-05-26T20:06:12.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-26T20:06:12.792 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:12.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-26T20:06:14.249 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-05-26T20:06:16.180 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-05-26T20:06:16.181 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-05-26T20:06:16.213 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.5tBnsm8LOB 2024-05-26T20:06:16.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5tBnsm8LOB 2024-05-26T20:06:16.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.5tBnsm8LOB 2024-05-26T20:06:16.336 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:16.362 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:16.431 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.5tBnsm8LOB is 6KB 2024-05-26T20:06:16.483 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.5tBnsm8LOB 2024-05-26T20:06:16.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-05-26T20:06:16.581 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-26T20:06:16.581 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-26T20:06:16.581 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-26T20:06:16.582 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.3.1.el8.x86_64.conf 2024-05-26T20:06:16.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:16.716 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:16.716 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-4.18.0-552.3.1.el8.x86_64 1/3 2024-05-26T20:06:16.716 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-4.18.0-552.3.1.el8.x86_64 2/3 2024-05-26T20:06:16.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-05-26T20:06:17.197 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-4.18.0-552.3.1.el8.x86_64 3/3 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:06:17.198 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:06:17.348 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T20:06:17.612 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.612 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.612 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.612 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-26T20:06:17.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-26T20:06:17.613 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-05-26T20:06:17.613 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-05-26T20:06:17.613 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T20:06:17.834 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-05-26T20:06:17.869 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.02 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 129.el8 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Group : System Environment/Base 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 9265586 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Relocations : (not relocatable) 2024-05-26T20:06:17.870 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : CentOS BuildSystem 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:06:17.871 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-05-26T20:06:17.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-26T20:06:17.872 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:17.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-26T20:06:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-05-26T20:06:20.856 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-05-26T20:06:20.858 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-05-26T20:06:20.892 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.6TXfhQDtI0 2024-05-26T20:06:20.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6TXfhQDtI0 2024-05-26T20:06:20.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.6TXfhQDtI0 2024-05-26T20:06:21.114 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.6TXfhQDtI0 is 6KB 2024-05-26T20:06:21.163 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.6TXfhQDtI0 2024-05-26T20:06:21.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-05-26T20:06:21.247 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-26T20:06:21.247 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-26T20:06:21.247 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-26T20:06:21.247 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.3.1.el8.x86_64.conf 2024-05-26T20:06:21.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.3.1.el8.x86_64 2024-05-26T20:06:21.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-05-26T20:06:46.787 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-05-26T20:06:46.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:06:46.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:06:51.458 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-05-26T20:06:51.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:06:51.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:07:05.247 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-05-26T20:07:09.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-05-26T20:07:14.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:07:14.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:07:17.311 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-05-26T20:07:18.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:07:18.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:07:29.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:07:29.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:08:01.224 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-05-26T20:08:01.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:08:01.463 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-552.3.1.el8.x86_64"... 2024-05-26T20:08:01.463 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-05-26T20:08:01.479 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-552.3.1.el8.x86_64 2024-05-26T20:08:01.479 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.3.1.el8.x86_64 vs 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:08:01.480 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-26T20:08:01.480 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-26T20:08:02.480 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-05-26T20:08:02.481 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-26T20:08:02.565 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-05-26T20:08:02.583 DEBUG:teuthology.parallel:result is None 2024-05-26T20:08:18.858 DEBUG:teuthology.orchestra.remote:timed out 2024-05-26T20:08:30.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:08:30.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:08:31.230 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-05-26T20:08:31.519 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:08:31.519 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-552.3.1.el8.x86_64"... 2024-05-26T20:08:31.520 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-05-26T20:08:31.539 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-552.3.1.el8.x86_64 2024-05-26T20:08:31.539 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.3.1.el8.x86_64 vs 4.18.0-552.3.1.el8.x86_64 2024-05-26T20:08:31.539 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-26T20:08:31.539 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-26T20:08:32.540 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-05-26T20:08:32.540 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-26T20:08:32.620 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-05-26T20:08:32.636 DEBUG:teuthology.parallel:result is None 2024-05-26T20:08:32.637 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-26T20:08:32.642 INFO:teuthology.task.internal:Creating test directory... 2024-05-26T20:08:32.642 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-26T20:08:32.645 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-26T20:08:32.667 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-26T20:08:32.685 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-26T20:08:32.729 INFO:teuthology.task.internal:Creating archive directory... 2024-05-26T20:08:32.730 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-26T20:08:32.732 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-26T20:08:32.782 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-26T20:08:32.789 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-26T20:08:32.789 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-26T20:08:32.814 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-26T20:08:32.854 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T20:08:32.855 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T20:08:32.866 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T20:08:32.867 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T20:08:32.868 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-26T20:08:32.876 INFO:teuthology.task.internal:Configuring sudo... 2024-05-26T20:08:32.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-26T20:08:32.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-26T20:08:32.941 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-26T20:08:32.950 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-26T20:08:32.950 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-26T20:08:32.983 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-26T20:08:33.002 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T20:08:33.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T20:08:33.153 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T20:08:33.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T20:08:33.241 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:08:33.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-26T20:08:33.309 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T20:08:33.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T20:08:33.411 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T20:08:33.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T20:08:33.542 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:08:33.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-26T20:08:33.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-05-26T20:08:33.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-05-26T20:08:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T20:08:33.722 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T20:08:34.096 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-26T20:08:34.103 INFO:teuthology.task.internal:Starting timer... 2024-05-26T20:08:34.103 INFO:teuthology.run_tasks:Running task pcp... 2024-05-26T20:08:34.147 INFO:teuthology.run_tasks:Running task selinux... 2024-05-26T20:08:34.180 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-05-26T20:08:34.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-05-26T20:08:34.250 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-05-26T20:08:34.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-05-26T20:08:34.337 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-05-26T20:08:34.339 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-26T20:08:34.339 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-05-26T20:08:34.392 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-05-26T20:08:34.393 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-05-26T20:08:34.457 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-05-26T20:08:34.457 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-05-26T20:08:34.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-26T20:08:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T20:08:34.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-26T20:08:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T20:08:34.515 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-26T20:08:34.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-05-26T20:08:34.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-05-26T20:08:34.582 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-26T20:08:34.592 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-26T20:08:34.602 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-26T20:08:34.603 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-26T20:14:48.793 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-05-26T20:14:48.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:14:48.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:14:48.880 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-05-26T20:14:48.956 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-05-26T20:14:48.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:14:48.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T20:14:49.043 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-05-26T20:14:49.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-05-26T20:14:49.114 INFO:teuthology.run_tasks:Running task clock... 2024-05-26T20:14:49.123 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-26T20:14:49.124 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-26T20:14:49.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T20:14:49.126 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-26T20:14:49.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T20:14:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-26T20:14:49.179 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-26T20:14:49.189 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-26T20:14:49.203 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-26T20:14:49.249 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-05-26T20:14:49.263 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-05-26T20:14:49.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-05-26T20:14:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-26T20:14:49.282 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-05-26T20:14:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-26T20:14:49.295 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-26T20:14:49.309 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-26T20:14:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.360 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.377 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.381 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T20:14:49.382 INFO:teuthology.run_tasks:Running task install... 2024-05-26T20:14:49.397 DEBUG:teuthology.task.install:project ceph 2024-05-26T20:14:49.397 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'}} 2024-05-26T20:14:49.397 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'} 2024-05-26T20:14:49.397 INFO:teuthology.task.install:Using flavor: default 2024-05-26T20:14:49.403 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-26T20:14:49.403 INFO:teuthology.task.install:extra packages: [] 2024-05-26T20:14:49.404 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'tag': None, 'wait_for_package': False} 2024-05-26T20:14:49.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=f1882d8093fb4b60c1d1e52444565787ca4bd6cc 2024-05-26T20:14:49.407 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'tag': None, 'wait_for_package': False} 2024-05-26T20:14:49.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=f1882d8093fb4b60c1d1e52444565787ca4bd6cc 2024-05-26T20:14:49.557 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/ 2024-05-26T20:14:49.557 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1355.gf1882d80 2024-05-26T20:14:49.635 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/ 2024-05-26T20:14:49.636 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1355.gf1882d80 2024-05-26T20:14:49.748 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-26T20:14:49.748 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:14:49.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-26T20:14:49.783 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-26T20:14:49.783 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-26T20:14:49.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-26T20:14:49.840 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:14:49.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-26T20:14:49.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-26T20:14:49.874 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-26T20:14:49.874 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f1882d8093fb4b60c1d1e52444565787ca4bd6cc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-26T20:14:49.942 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-26T20:14:49.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-26T20:14:50.016 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-05-26T20:14:50.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-05-26T20:14:50.023 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-26T20:14:50.089 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-05-26T20:14:50.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-05-26T20:14:50.295 INFO:teuthology.orchestra.run.smithi007.stdout:56 files removed 2024-05-26T20:14:50.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-05-26T20:14:50.350 INFO:teuthology.orchestra.run.smithi137.stdout:56 files removed 2024-05-26T20:14:50.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-05-26T20:14:51.134 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 209 kB/s | 80 kB 00:00 2024-05-26T20:14:51.208 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 188 kB/s | 80 kB 00:00 2024-05-26T20:14:51.535 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-05-26T20:14:51.639 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-05-26T20:14:51.870 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2024-05-26T20:14:52.018 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-05-26T20:14:58.519 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - AppStream 4.3 MB/s | 28 MB 00:06 2024-05-26T20:14:58.851 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 4.2 MB/s | 28 MB 00:06 2024-05-26T20:15:05.653 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - BaseOS 6.0 MB/s | 10 MB 00:01 2024-05-26T20:15:05.668 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 6.2 MB/s | 10 MB 00:01 2024-05-26T20:15:08.052 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 24 kB/s | 18 kB 00:00 2024-05-26T20:15:08.120 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras 26 kB/s | 18 kB 00:00 2024-05-26T20:15:08.654 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 8.0 kB 00:00 2024-05-26T20:15:08.828 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 8.0 kB 00:00 2024-05-26T20:15:10.151 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - PowerTools 3.6 MB/s | 5.0 MB 00:01 2024-05-26T20:15:10.246 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - PowerTools 3.7 MB/s | 5.0 MB 00:01 2024-05-26T20:15:11.357 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-05-26T20:15:11.372 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-05-26T20:15:12.602 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2024-05-26T20:15:12.833 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 10 MB/s | 14 MB 00:01 2024-05-26T20:15:17.066 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 405 kB/s | 24 kB 00:00 2024-05-26T20:15:17.240 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 419 kB/s | 24 kB 00:00 2024-05-26T20:15:20.929 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 13 M 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-05-26T20:15:20.932 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.7 M 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.6 M 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 5.5 M 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 22 M 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 24 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 781 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 474 k 2024-05-26T20:15:20.933 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 6.5 M 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 45 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 129 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 197 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 372 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 364 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 107 k 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:15:20.934 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout:Install 21 Packages 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 60 M 2024-05-26T20:15:20.935 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:15:21.035 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:15:21.038 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:15:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:15:21.038 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:15:21.038 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:15:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 13 M 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.7 M 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.6 M 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 5.5 M 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 22 M 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 24 k 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-26T20:15:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 781 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 474 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 6.5 M 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 45 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 129 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 197 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 372 k 2024-05-26T20:15:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 364 k 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 107 k 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout:Install 21 Packages 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:15:21.041 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 60 M 2024-05-26T20:15:21.042 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:15:23.457 INFO:teuthology.orchestra.run.smithi137.stdout:(1/23): ceph-base-18.2.2-1355.gf1882d80.el8.x86 2.5 MB/s | 5.5 MB 00:02 2024-05-26T20:15:23.532 INFO:teuthology.orchestra.run.smithi137.stdout:(2/23): ceph-selinux-18.2.2-1355.gf1882d80.el8. 325 kB/s | 24 kB 00:00 2024-05-26T20:15:23.898 INFO:teuthology.orchestra.run.smithi137.stdout:(3/23): libcephfs2-18.2.2-1355.gf1882d80.el8.x8 2.1 MB/s | 781 kB 00:00 2024-05-26T20:15:24.067 INFO:teuthology.orchestra.run.smithi007.stdout:(1/23): ceph-base-18.2.2-1355.gf1882d80.el8.x86 2.3 MB/s | 5.5 MB 00:02 2024-05-26T20:15:24.075 INFO:teuthology.orchestra.run.smithi137.stdout:(4/23): libradosstriper1-18.2.2-1355.gf1882d80. 2.6 MB/s | 474 kB 00:00 2024-05-26T20:15:24.151 INFO:teuthology.orchestra.run.smithi007.stdout:(2/23): ceph-selinux-18.2.2-1355.gf1882d80.el8. 291 kB/s | 24 kB 00:00 2024-05-26T20:15:24.737 INFO:teuthology.orchestra.run.smithi007.stdout:(3/23): libcephfs2-18.2.2-1355.gf1882d80.el8.x8 1.3 MB/s | 781 kB 00:00 2024-05-26T20:15:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:(4/23): libradosstriper1-18.2.2-1355.gf1882d80. 2.8 MB/s | 474 kB 00:00 2024-05-26T20:15:25.275 INFO:teuthology.orchestra.run.smithi137.stdout:(5/23): librgw2-18.2.2-1355.gf1882d80.el8.x86_6 5.5 MB/s | 6.5 MB 00:01 2024-05-26T20:15:25.371 INFO:teuthology.orchestra.run.smithi137.stdout:(6/23): python3-ceph-argparse-18.2.2-1355.gf188 468 kB/s | 45 kB 00:00 2024-05-26T20:15:25.459 INFO:teuthology.orchestra.run.smithi137.stdout:(7/23): python3-ceph-common-18.2.2-1355.gf1882d 1.4 MB/s | 129 kB 00:00 2024-05-26T20:15:25.551 INFO:teuthology.orchestra.run.smithi137.stdout:(8/23): python3-cephfs-18.2.2-1355.gf1882d80.el 2.1 MB/s | 197 kB 00:00 2024-05-26T20:15:25.626 INFO:teuthology.orchestra.run.smithi137.stdout:(9/23): python3-rados-18.2.2-1355.gf1882d80.el8 4.9 MB/s | 372 kB 00:00 2024-05-26T20:15:25.643 INFO:teuthology.orchestra.run.smithi007.stdout:(5/23): ceph-radosgw-18.2.2-1355.gf1882d80.el8. 3.3 MB/s | 13 MB 00:04 2024-05-26T20:15:25.710 INFO:teuthology.orchestra.run.smithi137.stdout:(10/23): python3-rbd-18.2.2-1355.gf1882d80.el8. 4.3 MB/s | 364 kB 00:00 2024-05-26T20:15:25.744 INFO:teuthology.orchestra.run.smithi007.stdout:(6/23): python3-ceph-argparse-18.2.2-1355.gf188 446 kB/s | 45 kB 00:00 2024-05-26T20:15:25.785 INFO:teuthology.orchestra.run.smithi137.stdout:(11/23): python3-rgw-18.2.2-1355.gf1882d80.el8. 1.4 MB/s | 107 kB 00:00 2024-05-26T20:15:25.828 INFO:teuthology.orchestra.run.smithi007.stdout:(7/23): python3-ceph-common-18.2.2-1355.gf1882d 1.5 MB/s | 129 kB 00:00 2024-05-26T20:15:25.936 INFO:teuthology.orchestra.run.smithi007.stdout:(8/23): python3-cephfs-18.2.2-1355.gf1882d80.el 1.8 MB/s | 197 kB 00:00 2024-05-26T20:15:25.986 INFO:teuthology.orchestra.run.smithi137.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 3.3 MB/s | 677 kB 00:00 2024-05-26T20:15:26.053 INFO:teuthology.orchestra.run.smithi007.stdout:(9/23): python3-rados-18.2.2-1355.gf1882d80.el8 3.1 MB/s | 372 kB 00:00 2024-05-26T20:15:26.119 INFO:teuthology.orchestra.run.smithi137.stdout:(13/23): ceph-radosgw-18.2.2-1355.gf1882d80.el8 2.7 MB/s | 13 MB 00:04 2024-05-26T20:15:26.145 INFO:teuthology.orchestra.run.smithi137.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-05-26T20:15:26.170 INFO:teuthology.orchestra.run.smithi007.stdout:(10/23): python3-rbd-18.2.2-1355.gf1882d80.el8. 3.1 MB/s | 364 kB 00:00 2024-05-26T20:15:26.220 INFO:teuthology.orchestra.run.smithi137.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 440 kB/s | 44 kB 00:00 2024-05-26T20:15:26.245 INFO:teuthology.orchestra.run.smithi007.stdout:(11/23): python3-rgw-18.2.2-1355.gf1882d80.el8. 1.4 MB/s | 107 kB 00:00 2024-05-26T20:15:26.295 INFO:teuthology.orchestra.run.smithi137.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 313 kB/s | 47 kB 00:00 2024-05-26T20:15:26.388 INFO:teuthology.orchestra.run.smithi007.stdout:(12/23): librgw2-18.2.2-1355.gf1882d80.el8.x86_ 4.4 MB/s | 6.5 MB 00:01 2024-05-26T20:15:26.613 INFO:teuthology.orchestra.run.smithi007.stdout:(13/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 677 kB 00:00 2024-05-26T20:15:26.663 INFO:teuthology.orchestra.run.smithi007.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 944 kB/s | 259 kB 00:00 2024-05-26T20:15:26.695 INFO:teuthology.orchestra.run.smithi137.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 147 kB/s | 59 kB 00:00 2024-05-26T20:15:26.722 INFO:teuthology.orchestra.run.smithi007.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 404 kB/s | 44 kB 00:00 2024-05-26T20:15:26.839 INFO:teuthology.orchestra.run.smithi007.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 269 kB/s | 47 kB 00:00 2024-05-26T20:15:26.904 INFO:teuthology.orchestra.run.smithi137.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 360 kB/s | 75 kB 00:00 2024-05-26T20:15:26.972 INFO:teuthology.orchestra.run.smithi007.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-05-26T20:15:27.006 INFO:teuthology.orchestra.run.smithi007.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 352 kB/s | 59 kB 00:00 2024-05-26T20:15:27.048 INFO:teuthology.orchestra.run.smithi007.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 996 kB/s | 75 kB 00:00 2024-05-26T20:15:27.179 INFO:teuthology.orchestra.run.smithi137.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 319 kB/s | 306 kB 00:00 2024-05-26T20:15:27.514 INFO:teuthology.orchestra.run.smithi007.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.4 MB/s | 1.7 MB 00:00 2024-05-26T20:15:28.090 INFO:teuthology.orchestra.run.smithi007.stdout:(21/23): librados2-18.2.2-1355.gf1882d80.el8.x8 3.6 MB/s | 3.7 MB 00:01 2024-05-26T20:15:28.205 INFO:teuthology.orchestra.run.smithi137.stdout:(20/23): librados2-18.2.2-1355.gf1882d80.el8.x8 3.6 MB/s | 3.7 MB 00:01 2024-05-26T20:15:28.432 INFO:teuthology.orchestra.run.smithi007.stdout:(22/23): ceph-common-18.2.2-1355.gf1882d80.el8. 3.2 MB/s | 22 MB 00:06 2024-05-26T20:15:28.788 INFO:teuthology.orchestra.run.smithi137.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 943 kB/s | 1.7 MB 00:01 2024-05-26T20:15:28.888 INFO:teuthology.orchestra.run.smithi137.stdout:(22/23): librbd1-18.2.2-1355.gf1882d80.el8.x86_ 5.2 MB/s | 3.6 MB 00:00 2024-05-26T20:15:30.907 INFO:teuthology.orchestra.run.smithi007.stdout:(23/23): librbd1-18.2.2-1355.gf1882d80.el8.x86_ 1.1 MB/s | 3.6 MB 00:03 2024-05-26T20:15:30.909 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:15:30.910 INFO:teuthology.orchestra.run.smithi007.stdout:Total 6.0 MB/s | 60 MB 00:09 2024-05-26T20:15:30.954 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:15:31.004 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:15:31.005 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:15:31.360 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:15:31.363 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:15:32.581 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:15:32.846 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-05-26T20:15:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-05-26T20:15:33.256 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-26T20:15:33.701 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-26T20:15:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:15:33.978 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:15:34.233 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 5/25 2024-05-26T20:15:34.406 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 6/25 2024-05-26T20:15:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 6/25 2024-05-26T20:15:35.012 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:15:35.081 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:15:35.303 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-26T20:15:35.521 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-26T20:15:35.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8. 10/25 2024-05-26T20:15:35.893 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 11/25 2024-05-26T20:15:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 12/25 2024-05-26T20:15:36.645 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 13/25 2024-05-26T20:15:36.708 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 13/25 2024-05-26T20:15:36.926 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-05-26T20:15:37.760 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-05-26T20:15:37.938 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 16/25 2024-05-26T20:15:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 16/25 2024-05-26T20:15:38.164 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 17/25 2024-05-26T20:15:38.321 INFO:teuthology.orchestra.run.smithi137.stdout:(23/23): ceph-common-18.2.2-1355.gf1882d80.el8. 1.3 MB/s | 22 MB 00:17 2024-05-26T20:15:38.323 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:15:38.323 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.4 MB/s | 60 MB 00:17 2024-05-26T20:15:38.366 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:15:38.386 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-26T20:15:38.413 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:15:38.413 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:15:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x8 19/25 2024-05-26T20:15:38.777 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:15:38.781 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:15:39.995 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:15:40.238 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-05-26T20:15:40.486 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-05-26T20:15:40.699 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-26T20:15:41.109 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-26T20:15:41.226 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:15:41.447 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:15:41.755 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 5/25 2024-05-26T20:15:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 6/25 2024-05-26T20:15:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 6/25 2024-05-26T20:15:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 21/25 2024-05-26T20:15:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 21/25 2024-05-26T20:15:42.591 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:15:42.660 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:15:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:15:42.900 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-26T20:15:43.105 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-26T20:15:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8. 10/25 2024-05-26T20:15:43.509 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 11/25 2024-05-26T20:15:43.758 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 12/25 2024-05-26T20:15:43.950 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 13/25 2024-05-26T20:15:44.011 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 13/25 2024-05-26T20:15:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-05-26T20:15:45.020 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-05-26T20:15:45.190 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 16/25 2024-05-26T20:15:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 16/25 2024-05-26T20:15:45.464 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 17/25 2024-05-26T20:15:45.650 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-26T20:15:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x8 19/25 2024-05-26T20:15:48.418 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:48.674 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 20/25 2024-05-26T20:15:49.573 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 21/25 2024-05-26T20:15:49.699 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 21/25 2024-05-26T20:15:49.915 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /run 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-05-26T20:16:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-05-26T20:16:02.992 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2024-05-26T20:16:02.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:04.435 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 23/25 2024-05-26T20:16:04.456 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 23/25 2024-05-26T20:16:04.457 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-26T20:16:04.703 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-26T20:16:04.703 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:05.422 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:05.422 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 1/25 2024-05-26T20:16:05.422 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 3/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 5/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 6/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8. 8/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x8 9/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 10/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 11/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 12/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 13/25 2024-05-26T20:16:05.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-26T20:16:05.424 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 24/25 2024-05-26T20:16:06.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:06.227 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:06.227 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-26T20:16:06.228 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:06.229 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:06.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-05-26T20:16:07.293 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:50 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:09.077 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:09.078 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:16:09.078 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 47 M 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 142 k 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:09.079 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 48 M 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 213 M 2024-05-26T20:16:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:16:09.211 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:16:09.211 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /run 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-05-26T20:16:09.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:09.382 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.7 MB/s | 203 kB 00:00 2024-05-26T20:16:09.440 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-05-26T20:16:09.466 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): libcephsqlite-18.2.2-1355.gf1882d80.el8. 707 kB/s | 142 kB 00:00 2024-05-26T20:16:09.491 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-05-26T20:16:09.725 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 273 kB/s | 71 kB 00:00 2024-05-26T20:16:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 23/25 2024-05-26T20:16:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 23/25 2024-05-26T20:16:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-26T20:16:10.871 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-26T20:16:10.871 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 1/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 3/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 4/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 5/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_6 6/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 7/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8. 8/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x8 9/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 10/25 2024-05-26T20:16:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 11/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 12/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 13/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 22/25 2024-05-26T20:16:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-26T20:16:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 24/25 2024-05-26T20:16:11.833 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-18.2.2-1355.gf1882d80.el8.x86_ 18 MB/s | 47 MB 00:02 2024-05-26T20:16:11.835 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:11.835 INFO:teuthology.orchestra.run.smithi007.stdout:Total 17 MB/s | 48 MB 00:02 2024-05-26T20:16:11.879 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:16:11.903 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:16:11.903 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:16:12.210 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:16:12.213 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:16:12.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-26T20:16:12.590 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:12.590 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-05-26T20:16:12.590 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.590 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-26T20:16:12.591 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:12.592 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:12.692 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:16:12.893 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-26T20:16:12.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-05-26T20:16:13.121 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-26T20:16:13.258 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-26T20:16:13.322 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-26T20:16:13.538 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-26T20:16:13.674 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 5/6 2024-05-26T20:16:13.687 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:56 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:15.357 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 47 M 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 142 k 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-26T20:16:15.358 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:15.359 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 48 M 2024-05-26T20:16:15.360 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 213 M 2024-05-26T20:16:15.360 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:16:15.607 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): libcephsqlite-18.2.2-1355.gf1882d80.el8. 731 kB/s | 142 kB 00:00 2024-05-26T20:16:15.632 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 924 kB/s | 203 kB 00:00 2024-05-26T20:16:15.757 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-05-26T20:16:15.816 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-05-26T20:16:15.842 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 804 kB/s | 188 kB 00:00 2024-05-26T20:16:17.933 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-18.2.2-1355.gf1882d80.el8.x86_ 19 MB/s | 47 MB 00:02 2024-05-26T20:16:17.935 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:17.935 INFO:teuthology.orchestra.run.smithi137.stdout:Total 19 MB/s | 48 MB 00:02 2024-05-26T20:16:17.976 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:16:17.997 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:16:17.997 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:16:18.296 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:16:18.298 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:16:18.526 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 5/6 2024-05-26T20:16:18.756 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 6/6 2024-05-26T20:16:18.786 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:16:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-26T20:16:19.041 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 6/6 2024-05-26T20:16:19.041 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 1/6 2024-05-26T20:16:19.041 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 2/6 2024-05-26T20:16:19.042 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-26T20:16:19.042 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-26T20:16:19.042 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-26T20:16:19.220 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-26T20:16:19.384 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-8.el8.x86_64 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:19.445 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-26T20:16:19.446 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-26T20:16:19.446 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-26T20:16:19.446 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:19.446 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:19.446 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-26T20:16:19.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-05-26T20:16:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-26T20:16:19.809 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 5/6 2024-05-26T20:16:20.420 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:03 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:22.216 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 6.4 k 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:16:22.220 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 2.3 M 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 1.6 M 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 246 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 4.8 M 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 18 M 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-26T20:16:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-26T20:16:22.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout:Install 32 Packages 2024-05-26T20:16:22.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:22.224 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 M 2024-05-26T20:16:22.224 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 124 M 2024-05-26T20:16:22.224 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:16:22.693 INFO:teuthology.orchestra.run.smithi007.stdout:(1/32): ceph-18.2.2-1355.gf1882d80.el8.x86_64.r 32 kB/s | 6.4 kB 00:00 2024-05-26T20:16:23.019 INFO:teuthology.orchestra.run.smithi007.stdout:(2/32): ceph-mgr-18.2.2-1355.gf1882d80.el8.x86_ 3.0 MB/s | 1.6 MB 00:00 2024-05-26T20:16:23.102 INFO:teuthology.orchestra.run.smithi007.stdout:(3/32): ceph-mds-18.2.2-1355.gf1882d80.el8.x86_ 3.7 MB/s | 2.3 MB 00:00 2024-05-26T20:16:23.178 INFO:teuthology.orchestra.run.smithi007.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1355.gf188 3.2 MB/s | 246 kB 00:00 2024-05-26T20:16:23.311 INFO:teuthology.orchestra.run.smithi007.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2024-05-26T20:16:23.362 INFO:teuthology.orchestra.run.smithi007.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 776 kB/s | 39 kB 00:00 2024-05-26T20:16:23.412 INFO:teuthology.orchestra.run.smithi007.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2024-05-26T20:16:23.479 INFO:teuthology.orchestra.run.smithi007.stdout:(8/32): ceph-mon-18.2.2-1355.gf1882d80.el8.x86_ 6.2 MB/s | 4.8 MB 00:00 2024-05-26T20:16:23.504 INFO:teuthology.orchestra.run.smithi007.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 620 kB/s | 57 kB 00:00 2024-05-26T20:16:23.680 INFO:teuthology.orchestra.run.smithi007.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.4 MB/s | 249 kB 00:00 2024-05-26T20:16:23.705 INFO:teuthology.orchestra.run.smithi007.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2024-05-26T20:16:23.756 INFO:teuthology.orchestra.run.smithi007.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 8.3 MB/s | 641 kB 00:00 2024-05-26T20:16:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 725 kB/s | 109 kB 00:00 2024-05-26T20:16:24.098 INFO:teuthology.orchestra.run.smithi007.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 127 kB/s | 44 kB 00:00 2024-05-26T20:16:24.323 INFO:teuthology.orchestra.run.smithi007.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 768 kB/s | 173 kB 00:00 2024-05-26T20:16:24.349 INFO:teuthology.orchestra.run.smithi007.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 376 kB/s | 185 kB 00:00 2024-05-26T20:16:24.362 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 5/6 2024-05-26T20:16:24.532 INFO:teuthology.orchestra.run.smithi007.stdout:(17/32): ceph-osd-18.2.2-1355.gf1882d80.el8.x86 12 MB/s | 18 MB 00:01 2024-05-26T20:16:24.557 INFO:teuthology.orchestra.run.smithi007.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 53 kB/s | 11 kB 00:00 2024-05-26T20:16:24.583 INFO:teuthology.orchestra.run.smithi007.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-05-26T20:16:24.615 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 6/6 2024-05-26T20:16:24.616 INFO:teuthology.orchestra.run.smithi007.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2024-05-26T20:16:24.692 INFO:teuthology.orchestra.run.smithi007.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2024-05-26T20:16:24.795 INFO:teuthology.orchestra.run.smithi007.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 152 kB/s | 16 kB 00:00 2024-05-26T20:16:24.825 INFO:teuthology.orchestra.run.smithi007.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-05-26T20:16:24.876 INFO:teuthology.orchestra.run.smithi007.stdout:(24/32): python3-logutils-0.3.5-11.el8.noarch.r 153 kB/s | 49 kB 00:00 2024-05-26T20:16:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 177 kB/s | 19 kB 00:00 2024-05-26T20:16:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 6/6 2024-05-26T20:16:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 1/6 2024-05-26T20:16:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 2/6 2024-05-26T20:16:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-26T20:16:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-26T20:16:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-26T20:16:24.927 INFO:teuthology.orchestra.run.smithi007.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-05-26T20:16:24.960 INFO:teuthology.orchestra.run.smithi007.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-05-26T20:16:25.002 INFO:teuthology.orchestra.run.smithi007.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 367 kB/s | 27 kB 00:00 2024-05-26T20:16:25.028 INFO:teuthology.orchestra.run.smithi007.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-05-26T20:16:25.086 INFO:teuthology.orchestra.run.smithi007.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-05-26T20:16:25.120 INFO:teuthology.orchestra.run.smithi007.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-05-26T20:16:25.199 INFO:teuthology.orchestra.run.smithi007.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2024-05-26T20:16:25.203 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:25.203 INFO:teuthology.orchestra.run.smithi007.stdout:Total 10 MB/s | 31 MB 00:02 2024-05-26T20:16:25.288 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:16:25.329 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:16:25.330 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-8.el8.x86_64 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-26T20:16:25.362 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-26T20:16:25.363 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-26T20:16:25.363 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:25.363 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:25.648 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:16:25.652 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:16:25.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-05-26T20:16:26.419 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:09 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:26.798 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:16:27.023 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-26T20:16:27.255 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-26T20:16:27.452 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-26T20:16:27.697 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-26T20:16:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-26T20:16:28.090 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:16:28.093 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 6.4 k 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 2.3 M 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 1.6 M 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 246 k 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 4.8 M 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 18 M 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-26T20:16:28.094 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-26T20:16:28.095 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:16:28.096 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:28.097 INFO:teuthology.orchestra.run.smithi137.stdout:Install 32 Packages 2024-05-26T20:16:28.097 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:28.097 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 M 2024-05-26T20:16:28.097 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 124 M 2024-05-26T20:16:28.097 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:16:28.109 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-26T20:16:28.284 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-26T20:16:28.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-26T20:16:28.628 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-18.2.2-1355.gf1882d80.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2024-05-26T20:16:28.689 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-26T20:16:28.838 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-mgr-18.2.2-1355.gf1882d80.el8.x86_ 4.7 MB/s | 1.6 MB 00:00 2024-05-26T20:16:28.942 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-26T20:16:29.156 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-26T20:16:29.220 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-mon-18.2.2-1355.gf1882d80.el8.x86_ 8.2 MB/s | 4.8 MB 00:00 2024-05-26T20:16:29.262 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-mds-18.2.2-1355.gf1882d80.el8.x86_ 3.0 MB/s | 2.3 MB 00:00 2024-05-26T20:16:29.287 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): ceph-mgr-modules-core-18.2.2-1355.gf188 3.6 MB/s | 246 kB 00:00 2024-05-26T20:16:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-26T20:16:29.546 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2024-05-26T20:16:29.738 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 540 kB/s | 103 kB 00:00 2024-05-26T20:16:29.763 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 313 kB/s | 157 kB 00:00 2024-05-26T20:16:29.808 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-26T20:16:29.939 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): ceph-osd-18.2.2-1355.gf1882d80.el8.x86_ 17 MB/s | 18 MB 00:01 2024-05-26T20:16:29.964 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 253 kB/s | 57 kB 00:00 2024-05-26T20:16:30.056 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-05-26T20:16:30.084 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-26T20:16:30.131 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-05-26T20:16:30.165 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 3.1 MB/s | 641 kB 00:00 2024-05-26T20:16:30.199 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 765 kB/s | 109 kB 00:00 2024-05-26T20:16:30.246 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-26T20:16:30.457 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 134 kB/s | 44 kB 00:00 2024-05-26T20:16:30.484 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-26T20:16:30.566 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 462 kB/s | 185 kB 00:00 2024-05-26T20:16:30.674 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 363 kB/s | 173 kB 00:00 2024-05-26T20:16:30.681 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-26T20:16:30.775 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 53 kB/s | 11 kB 00:00 2024-05-26T20:16:30.925 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 70 kB/s | 18 kB 00:00 2024-05-26T20:16:30.950 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 781 kB/s | 384 kB 00:00 2024-05-26T20:16:31.016 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-26T20:16:31.100 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 150 kB/s | 49 kB 00:00 2024-05-26T20:16:31.237 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-26T20:16:31.251 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 180 kB/s | 59 kB 00:00 2024-05-26T20:16:31.359 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 60 kB/s | 16 kB 00:00 2024-05-26T20:16:31.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-26T20:16:31.443 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 574 kB/s | 283 kB 00:00 2024-05-26T20:16:31.535 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 66 kB/s | 19 kB 00:00 2024-05-26T20:16:31.610 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 95 kB/s | 24 kB 00:00 2024-05-26T20:16:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-26T20:16:31.711 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 109 kB/s | 29 kB 00:00 2024-05-26T20:16:31.811 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 100 kB/s | 27 kB 00:00 2024-05-26T20:16:31.855 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-26T20:16:32.104 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 519 kB/s | 255 kB 00:00 2024-05-26T20:16:32.111 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-26T20:16:32.145 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 256 kB/s | 85 kB 00:00 2024-05-26T20:16:32.262 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 456 kB/s | 251 kB 00:00 2024-05-26T20:16:32.341 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-26T20:16:32.368 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 87 kB/s | 23 kB 00:00 2024-05-26T20:16:32.371 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:32.372 INFO:teuthology.orchestra.run.smithi137.stdout:Total 7.2 MB/s | 31 MB 00:04 2024-05-26T20:16:32.454 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:16:32.491 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:16:32.492 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:16:32.660 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-26T20:16:32.818 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:16:32.821 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:16:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-26T20:16:33.544 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8. 27/32 2024-05-26T20:16:33.713 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 28/32 2024-05-26T20:16:33.994 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:16:34.253 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-26T20:16:34.481 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-26T20:16:34.697 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-26T20:16:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-26T20:16:35.253 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-26T20:16:35.457 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 28/32 2024-05-26T20:16:35.492 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-26T20:16:35.688 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 29/32 2024-05-26T20:16:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-26T20:16:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-26T20:16:36.089 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-26T20:16:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 29/32 2024-05-26T20:16:36.351 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-26T20:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 30/32 2024-05-26T20:16:36.563 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-26T20:16:36.629 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 30/32 2024-05-26T20:16:36.804 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-26T20:16:36.812 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 31/32 2024-05-26T20:16:36.831 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 31/32 2024-05-26T20:16:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 32/32 2024-05-26T20:16:37.231 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-26T20:16:37.387 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 32/32 2024-05-26T20:16:37.387 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 1/32 2024-05-26T20:16:37.387 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 2/32 2024-05-26T20:16:37.387 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 3/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 4/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 5/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8. 6/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-26T20:16:37.388 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-26T20:16:37.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-26T20:16:37.476 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-26T20:16:37.681 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-26T20:16:37.956 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-26T20:16:38.216 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-26T20:16:38.541 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-26T20:16:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-2.6-1.el8.noarch 2024-05-26T20:16:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:38.658 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:38.745 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-26T20:16:38.933 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-26T20:16:39.119 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-26T20:16:39.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-05-26T20:16:39.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-26T20:16:39.593 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-26T20:16:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-26T20:16:39.971 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:22 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:40.201 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-26T20:16:40.530 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-26T20:16:40.903 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8. 27/32 2024-05-26T20:16:41.089 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 28/32 2024-05-26T20:16:41.702 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:16:41.755 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:41.757 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:16:41.757 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:41.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-05-26T20:16:42.642 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:25 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 28/32 2024-05-26T20:16:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 29/32 2024-05-26T20:16:43.444 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 29/32 2024-05-26T20:16:44.172 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 30/32 2024-05-26T20:16:44.401 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 30/32 2024-05-26T20:16:44.430 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 173 k 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:16:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:44.432 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 173 k 2024-05-26T20:16:44.432 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 692 k 2024-05-26T20:16:44.432 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:16:44.614 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 31/32 2024-05-26T20:16:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 31/32 2024-05-26T20:16:44.729 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-18.2.2-1355.gf1882d80.el8.noarch.rpm 581 kB/s | 173 kB 00:00 2024-05-26T20:16:44.730 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:44.730 INFO:teuthology.orchestra.run.smithi007.stdout:Total 578 kB/s | 173 kB 00:00 2024-05-26T20:16:44.730 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:16:44.739 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:16:44.739 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:16:44.742 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:16:44.745 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:16:44.747 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 32/32 2024-05-26T20:16:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 32/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 1/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 2/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 3/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 4/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 5/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8. 6/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-26T20:16:45.158 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-26T20:16:45.159 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-26T20:16:45.160 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-26T20:16:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:46.155 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:46.166 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-26T20:16:46.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:46.166 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:16:46.166 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:46.166 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-26T20:16:46.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-2.6-1.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-26T20:16:46.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:46.169 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:46.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-05-26T20:16:46.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-05-26T20:16:47.191 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:30 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:47.225 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:48.827 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:16:48.874 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:16:48.875 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:16:48.876 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:48.991 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:48.991 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 166 k 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 166 k 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 503 k 2024-05-26T20:16:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:16:49.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-05-26T20:16:49.599 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-18.2.2-1355.gf1882d 273 kB/s | 166 kB 00:00 2024-05-26T20:16:49.600 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:49.600 INFO:teuthology.orchestra.run.smithi007.stdout:Total 272 kB/s | 166 kB 00:00 2024-05-26T20:16:49.600 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:16:49.614 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:16:49.614 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:16:49.671 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:16:49.674 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:16:49.750 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:32 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:50.364 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:16:51.069 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:51.394 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:16:51.395 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:51.395 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:16:51.395 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 173 k 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 173 k 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 692 k 2024-05-26T20:16:51.396 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:16:51.511 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:51.702 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-18.2.2-1355.gf1882d80.el8.noarch.rpm 565 kB/s | 173 kB 00:00 2024-05-26T20:16:51.702 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:51.703 INFO:teuthology.orchestra.run.smithi137.stdout:Total 563 kB/s | 173 kB 00:00 2024-05-26T20:16:51.703 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:16:51.710 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:16:51.711 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:16:51.713 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:16:51.716 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:16:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:16:51.913 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:51.913 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:16:51.913 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:51.914 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:52.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-05-26T20:16:52.425 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:52.662 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:53.044 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/1 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:53.047 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:53.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-05-26T20:16:54.096 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:54.779 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:16:54.831 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:54.833 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:16:54.833 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:16:54.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-05-26T20:16:55.731 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:38 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:55.769 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 166 k 2024-05-26T20:16:55.770 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 166 k 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 503 k 2024-05-26T20:16:55.771 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:16:56.232 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-18.2.2-1355.gf1882d 360 kB/s | 166 kB 00:00 2024-05-26T20:16:56.232 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:56.232 INFO:teuthology.orchestra.run.smithi137.stdout:Total 359 kB/s | 166 kB 00:00 2024-05-26T20:16:56.233 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:16:56.244 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:16:56.244 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:16:56.293 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:16:56.296 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:16:56.409 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:16:56.612 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:57.032 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:57.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80. 1/1 2024-05-26T20:16:57.387 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:57.387 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:16:57.388 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:16:57.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:16:57.388 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:16:57.588 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 3.5 M 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 24 k 2024-05-26T20:16:57.589 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 16 k 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2024-05-26T20:16:57.590 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:16:57.591 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.8 M 2024-05-26T20:16:57.591 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 87 M 2024-05-26T20:16:57.591 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:16:57.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-05-26T20:16:57.807 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1355.gf188 85 kB/s | 16 kB 00:00 2024-05-26T20:16:57.908 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1355.gf18 83 kB/s | 24 kB 00:00 2024-05-26T20:16:58.091 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 118 kB/s | 34 kB 00:00 2024-05-26T20:16:58.341 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 453 kB/s | 196 kB 00:00 2024-05-26T20:16:58.470 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:16:58.727 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1355.gf1882d80 3.2 MB/s | 3.5 MB 00:01 2024-05-26T20:16:58.731 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:16:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.3 MB/s | 3.8 MB 00:01 2024-05-26T20:16:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:16:58.741 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:16:58.741 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:16:58.835 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:16:58.838 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:16:59.141 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:16:59.317 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-26T20:16:59.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-26T20:16:59.649 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.n 3/5 2024-05-26T20:17:00.086 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:17:00.134 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:00.135 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:17:00.136 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:00.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-05-26T20:17:00.848 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8. 4/5 2024-05-26T20:17:01.007 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:44 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 5/5 2024-05-26T20:17:01.404 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 5/5 2024-05-26T20:17:01.404 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8. 1/5 2024-05-26T20:17:01.404 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 2/5 2024-05-26T20:17:01.404 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.n 3/5 2024-05-26T20:17:01.405 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-26T20:17:01.950 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:01.951 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:02.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-26T20:17:02.638 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 3.5 M 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 24 k 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 16 k 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:Install 5 Packages 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.8 M 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 87 M 2024-05-26T20:17:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:02.869 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:45 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:02.919 INFO:teuthology.orchestra.run.smithi137.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1355.gf18 96 kB/s | 24 kB 00:00 2024-05-26T20:17:02.944 INFO:teuthology.orchestra.run.smithi137.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1355.gf188 59 kB/s | 16 kB 00:00 2024-05-26T20:17:03.036 INFO:teuthology.orchestra.run.smithi137.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 288 kB/s | 34 kB 00:00 2024-05-26T20:17:03.086 INFO:teuthology.orchestra.run.smithi137.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2024-05-26T20:17:03.270 INFO:teuthology.orchestra.run.smithi137.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1355.gf1882d80 5.8 MB/s | 3.5 MB 00:00 2024-05-26T20:17:03.271 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:03.271 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.0 MB/s | 3.8 MB 00:00 2024-05-26T20:17:03.272 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:03.279 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:03.279 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:03.358 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:03.362 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:03.616 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:03.833 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-26T20:17:04.030 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-26T20:17:04.192 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.n 3/5 2024-05-26T20:17:04.643 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 7.4 M 2024-05-26T20:17:04.644 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-26T20:17:04.645 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:04.646 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:04.646 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-05-26T20:17:04.646 INFO:teuthology.orchestra.run.smithi007.stdout:Install 8 Packages 2024-05-26T20:17:04.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:04.647 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 36 M 2024-05-26T20:17:04.647 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 210 M 2024-05-26T20:17:04.647 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:05.311 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8. 4/5 2024-05-26T20:17:05.526 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 5/5 2024-05-26T20:17:05.624 INFO:teuthology.orchestra.run.smithi007.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.2-135 10 MB/s | 7.4 MB 00:00 2024-05-26T20:17:05.826 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 5/5 2024-05-26T20:17:05.826 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8. 1/5 2024-05-26T20:17:05.826 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarc 2/5 2024-05-26T20:17:05.827 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.n 3/5 2024-05-26T20:17:05.827 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:06.456 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:06.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-26T20:17:06.716 INFO:teuthology.orchestra.run.smithi007.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 2.6 MB/s | 4.7 MB 00:01 2024-05-26T20:17:06.866 INFO:teuthology.orchestra.run.smithi007.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.5 MB/s | 225 kB 00:00 2024-05-26T20:17:06.967 INFO:teuthology.orchestra.run.smithi007.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.3 MB/s | 4.9 MB 00:02 2024-05-26T20:17:07.067 INFO:teuthology.orchestra.run.smithi007.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.6 MB/s | 3.7 MB 00:01 2024-05-26T20:17:07.474 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:50 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:07.495 INFO:teuthology.orchestra.run.smithi007.stdout:(6/8): libgfortran-8.5.0-22.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-05-26T20:17:07.787 INFO:teuthology.orchestra.run.smithi007.stdout:(7/8): libquadmath-8.5.0-22.el8.x86_64.rpm 240 kB/s | 172 kB 00:00 2024-05-26T20:17:09.129 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 7.4 M 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:09.130 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout:Install 8 Packages 2024-05-26T20:17:09.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:09.132 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 36 M 2024-05-26T20:17:09.132 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 210 M 2024-05-26T20:17:09.132 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:10.059 INFO:teuthology.orchestra.run.smithi137.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.2-135 14 MB/s | 7.4 MB 00:00 2024-05-26T20:17:10.692 INFO:teuthology.orchestra.run.smithi137.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.9 MB/s | 3.7 MB 00:00 2024-05-26T20:17:10.784 INFO:teuthology.orchestra.run.smithi137.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.8 MB/s | 4.9 MB 00:01 2024-05-26T20:17:10.818 INFO:teuthology.orchestra.run.smithi137.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2024-05-26T20:17:10.977 INFO:teuthology.orchestra.run.smithi137.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 3.2 MB/s | 4.7 MB 00:01 2024-05-26T20:17:11.112 INFO:teuthology.orchestra.run.smithi137.stdout:(6/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.1 MB/s | 645 kB 00:00 2024-05-26T20:17:11.169 INFO:teuthology.orchestra.run.smithi137.stdout:(7/8): libquadmath-8.5.0-22.el8.x86_64.rpm 899 kB/s | 172 kB 00:00 2024-05-26T20:17:14.186 INFO:teuthology.orchestra.run.smithi007.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.9 MB/s | 14 MB 00:07 2024-05-26T20:17:14.187 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:14.187 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.8 MB/s | 36 MB 00:09 2024-05-26T20:17:14.429 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:14.449 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:14.449 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:15.121 INFO:teuthology.orchestra.run.smithi137.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 3.3 MB/s | 14 MB 00:04 2024-05-26T20:17:15.122 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:15.122 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.0 MB/s | 36 MB 00:05 2024-05-26T20:17:15.355 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:15.373 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:15.373 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:15.464 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:15.614 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-26T20:17:15.751 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-26T20:17:15.808 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:15.811 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:15.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-26T20:17:16.380 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:16.516 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-26T20:17:16.564 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-26T20:17:16.685 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-26T20:17:16.764 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:16.842 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-26T20:17:17.420 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:17.459 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-26T20:17:17.547 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-26T20:17:18.008 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:18.273 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-26T20:17:18.605 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-26T20:17:18.609 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:18.791 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-26T20:17:19.012 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-26T20:17:19.469 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-26T20:17:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-26T20:17:20.131 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-26T20:17:20.902 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-26T20:17:21.889 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-26T20:17:22.221 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-26T20:17:22.387 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 8/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 8/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 1/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-26T20:17:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-26T20:17:23.144 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-26T20:17:23.162 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:23.163 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:23.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-05-26T20:17:23.365 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 8/8 2024-05-26T20:17:23.686 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 8/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d8 1/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-26T20:17:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-26T20:17:24.084 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:07 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:24.097 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-26T20:17:24.097 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:24.097 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:24.098 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:24.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-05-26T20:17:25.021 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:08 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:25.871 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 49 k 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-26T20:17:25.873 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:25.874 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout:Install 14 Packages 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 25 M 2024-05-26T20:17:25.875 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:26.418 INFO:teuthology.orchestra.run.smithi007.stdout:(1/14): ceph-mgr-rook-18.2.2-1355.gf1882d80.el8 295 kB/s | 49 kB 00:00 2024-05-26T20:17:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:(2/14): python3-jsonpointer-1.10-11.el8.noarch. 70 kB/s | 20 kB 00:00 2024-05-26T20:17:26.560 INFO:teuthology.orchestra.run.smithi007.stdout:(3/14): python3-jsonpatch-1.21-2.el8.noarch.rpm 86 kB/s | 27 kB 00:00 2024-05-26T20:17:26.662 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 49 k 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-26T20:17:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:26.665 INFO:teuthology.orchestra.run.smithi137.stdout:Install 14 Packages 2024-05-26T20:17:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:26.666 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-05-26T20:17:26.666 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 25 M 2024-05-26T20:17:26.666 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:26.686 INFO:teuthology.orchestra.run.smithi007.stdout:(4/14): python3-jwt-1.6.1-2.el8.noarch.rpm 341 kB/s | 43 kB 00:00 2024-05-26T20:17:26.761 INFO:teuthology.orchestra.run.smithi007.stdout:(5/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-05-26T20:17:26.803 INFO:teuthology.orchestra.run.smithi007.stdout:(6/14): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2024-05-26T20:17:26.853 INFO:teuthology.orchestra.run.smithi007.stdout:(7/14): python3-pyasn1-modules-0.3.7-6.el8.noar 346 kB/s | 110 kB 00:00 2024-05-26T20:17:26.879 INFO:teuthology.orchestra.run.smithi007.stdout:(8/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 275 kB/s | 126 kB 00:00 2024-05-26T20:17:27.087 INFO:teuthology.orchestra.run.smithi007.stdout:(9/14): python3-cachetools-3.1.1-4.el8.noarch.r 116 kB/s | 33 kB 00:00 2024-05-26T20:17:27.104 INFO:teuthology.orchestra.run.smithi007.stdout:(10/14): python3-certifi-2018.10.15-7.el8.noarc 65 kB/s | 16 kB 00:00 2024-05-26T20:17:27.203 INFO:teuthology.orchestra.run.smithi137.stdout:(1/14): python3-jsonpointer-1.10-11.el8.noarch. 161 kB/s | 20 kB 00:00 2024-05-26T20:17:27.228 INFO:teuthology.orchestra.run.smithi137.stdout:(2/14): python3-jsonpatch-1.21-2.el8.noarch.rpm 178 kB/s | 27 kB 00:00 2024-05-26T20:17:27.263 INFO:teuthology.orchestra.run.smithi007.stdout:(11/14): python3-google-auth-1.1.1-10.el8.noarc 211 kB/s | 81 kB 00:00 2024-05-26T20:17:27.319 INFO:teuthology.orchestra.run.smithi137.stdout:(3/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-05-26T20:17:27.338 INFO:teuthology.orchestra.run.smithi007.stdout:(12/14): python3-rsa-4.9-2.el8.noarch.rpm 275 kB/s | 64 kB 00:00 2024-05-26T20:17:27.345 INFO:teuthology.orchestra.run.smithi137.stdout:(4/14): ceph-mgr-rook-18.2.2-1355.gf1882d80.el8 184 kB/s | 49 kB 00:00 2024-05-26T20:17:27.362 INFO:teuthology.orchestra.run.smithi137.stdout:(5/14): python3-pyasn1-modules-0.3.7-6.el8.noar 821 kB/s | 110 kB 00:00 2024-05-26T20:17:27.397 INFO:teuthology.orchestra.run.smithi007.stdout:(13/14): python3-websocket-client-0.56.0-5.el8. 459 kB/s | 61 kB 00:00 2024-05-26T20:17:27.470 INFO:teuthology.orchestra.run.smithi137.stdout:(6/14): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2024-05-26T20:17:27.496 INFO:teuthology.orchestra.run.smithi137.stdout:(7/14): python3-requests-oauthlib-1.0.0-1.el8.n 319 kB/s | 43 kB 00:00 2024-05-26T20:17:27.513 INFO:teuthology.orchestra.run.smithi137.stdout:(8/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 922 kB/s | 155 kB 00:00 2024-05-26T20:17:27.605 INFO:teuthology.orchestra.run.smithi137.stdout:(9/14): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2024-05-26T20:17:27.630 INFO:teuthology.orchestra.run.smithi137.stdout:(10/14): python3-cachetools-3.1.1-4.el8.noarch. 207 kB/s | 33 kB 00:00 2024-05-26T20:17:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:(14/14): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-05-26T20:17:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-05-26T20:17:27.714 INFO:teuthology.orchestra.run.smithi137.stdout:(11/14): python3-google-auth-1.1.1-10.el8.noarc 405 kB/s | 81 kB 00:00 2024-05-26T20:17:27.726 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:27.731 INFO:teuthology.orchestra.run.smithi137.stdout:(12/14): python3-rsa-4.9-2.el8.noarch.rpm 640 kB/s | 64 kB 00:00 2024-05-26T20:17:27.743 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:27.743 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:27.798 INFO:teuthology.orchestra.run.smithi137.stdout:(13/14): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2024-05-26T20:17:27.837 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:27.840 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:27.874 INFO:teuthology.orchestra.run.smithi137.stdout:(14/14): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2024-05-26T20:17:27.874 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:27.874 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-05-26T20:17:27.941 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:27.957 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:27.957 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:28.044 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:28.047 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:28.374 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:28.601 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:28.641 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-05-26T20:17:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-05-26T20:17:28.871 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/14 2024-05-26T20:17:29.031 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/14 2024-05-26T20:17:29.099 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/14 2024-05-26T20:17:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/14 2024-05-26T20:17:29.352 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/14 2024-05-26T20:17:29.471 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/14 2024-05-26T20:17:29.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/14 2024-05-26T20:17:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/14 2024-05-26T20:17:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/14 2024-05-26T20:17:29.933 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/14 2024-05-26T20:17:30.198 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/14 2024-05-26T20:17:30.233 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/14 2024-05-26T20:17:30.501 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-05-26T20:17:30.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-05-26T20:17:30.744 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-05-26T20:17:30.751 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-05-26T20:17:31.786 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-05-26T20:17:31.874 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-05-26T20:17:32.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 11/14 2024-05-26T20:17:32.151 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 11/14 2024-05-26T20:17:32.296 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 12/14 2024-05-26T20:17:32.344 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 12/14 2024-05-26T20:17:32.525 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-05-26T20:17:32.557 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-05-26T20:17:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 14/14 2024-05-26T20:17:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 14/14 2024-05-26T20:17:33.061 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 14/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 1/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 9/14 2024-05-26T20:17:33.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-05-26T20:17:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 11/14 2024-05-26T20:17:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 12/14 2024-05-26T20:17:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 13/14 2024-05-26T20:17:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 14/14 2024-05-26T20:17:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 1/14 2024-05-26T20:17:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/14 2024-05-26T20:17:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 9/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 11/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 12/14 2024-05-26T20:17:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 13/14 2024-05-26T20:17:33.610 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 14/14 2024-05-26T20:17:33.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-26T20:17:33.611 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-26T20:17:33.612 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-26T20:17:33.612 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:33.612 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:33.642 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 14/14 2024-05-26T20:17:33.642 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-26T20:17:33.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-26T20:17:33.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-26T20:17:33.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-26T20:17:33.644 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:33.644 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:33.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-05-26T20:17:33.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-05-26T20:17:34.506 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:17 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:34.528 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:17 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:36.174 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:36.175 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:36.175 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 138 k 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.10.1-5.el8 appstream 579 k 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout:Install 5 Packages 2024-05-26T20:17:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:36.177 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.0 M 2024-05-26T20:17:36.177 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-05-26T20:17:36.177 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:36.319 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 138 k 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.10.1-5.el8 appstream 579 k 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:36.321 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:36.322 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:36.322 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2024-05-26T20:17:36.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:36.322 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.0 M 2024-05-26T20:17:36.322 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-05-26T20:17:36.323 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:36.589 INFO:teuthology.orchestra.run.smithi137.stdout:(1/5): python3-jinja2-2.10.1-5.el8.noarch.rpm 3.1 MB/s | 579 kB 00:00 2024-05-26T20:17:36.699 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): python3-jinja2-2.10.1-5.el8.noarch.rpm 2.0 MB/s | 579 kB 00:00 2024-05-26T20:17:36.866 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1355.gf1882d80.e 304 kB/s | 138 kB 00:00 2024-05-26T20:17:36.891 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.4 MB/s | 476 kB 00:00 2024-05-26T20:17:36.940 INFO:teuthology.orchestra.run.smithi137.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1355.gf1882d80.e 261 kB/s | 138 kB 00:00 2024-05-26T20:17:36.983 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 515 kB/s | 60 kB 00:00 2024-05-26T20:17:37.215 INFO:teuthology.orchestra.run.smithi137.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 761 kB/s | 476 kB 00:00 2024-05-26T20:17:37.274 INFO:teuthology.orchestra.run.smithi137.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 180 kB/s | 60 kB 00:00 2024-05-26T20:17:37.790 INFO:teuthology.orchestra.run.smithi137.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.5 MB/s | 4.8 MB 00:01 2024-05-26T20:17:37.791 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:37.792 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.7 MB/s | 6.0 MB 00:01 2024-05-26T20:17:37.846 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:37.857 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:37.857 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:37.965 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:37.968 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:38.236 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:38.550 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 2.2 MB/s | 4.8 MB 00:02 2024-05-26T20:17:38.551 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:38.551 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.7 MB/s | 6.0 MB 00:02 2024-05-26T20:17:38.607 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:38.734 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:38.737 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:39.337 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-05-26T20:17:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-05-26T20:17:40.492 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-05-26T20:17:40.550 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-05-26T20:17:40.883 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-05-26T20:17:40.939 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-05-26T20:17:41.195 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.10.1-5.el8.noarch 4/5 2024-05-26T20:17:41.237 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.10.1-5.el8.noarch 4/5 2024-05-26T20:17:41.451 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 5/5 2024-05-26T20:17:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 5/5 2024-05-26T20:17:41.736 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 5/5 2024-05-26T20:17:41.736 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/5 2024-05-26T20:17:41.737 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-05-26T20:17:41.737 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.10.1-5.el8.noarch 3/5 2024-05-26T20:17:41.737 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-05-26T20:17:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 5/5 2024-05-26T20:17:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 1/5 2024-05-26T20:17:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-05-26T20:17:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.10.1-5.el8.noarch 3/5 2024-05-26T20:17:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-26T20:17:42.125 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.10.1-5.el8.noarch 2024-05-26T20:17:42.126 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-26T20:17:42.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:42.126 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.10.1-5.el8.noarch 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:42.194 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:42.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-05-26T20:17:42.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-05-26T20:17:43.009 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:26 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:43.096 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:26 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:44.662 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:44.662 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:44.662 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 859 k 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 942 k 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 3.0 M 2024-05-26T20:17:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:44.841 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 660 kB/s | 83 kB 00:00 2024-05-26T20:17:44.862 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:44.863 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:17:44.863 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 859 k 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-05-26T20:17:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:44.865 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 942 k 2024-05-26T20:17:44.865 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 3.0 M 2024-05-26T20:17:44.865 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:45.050 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 634 kB/s | 83 kB 00:00 2024-05-26T20:17:45.200 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-18.2.2-1355.gf1882d80.el8.x86_ 1.7 MB/s | 859 kB 00:00 2024-05-26T20:17:45.200 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:45.200 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.7 MB/s | 942 kB 00:00 2024-05-26T20:17:45.208 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-18.2.2-1355.gf1882d80.el8.x86_ 2.9 MB/s | 859 kB 00:00 2024-05-26T20:17:45.209 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:45.209 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.7 MB/s | 942 kB 00:00 2024-05-26T20:17:45.210 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:45.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:45.244 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:45.244 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:45.318 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:45.321 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:45.499 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-26T20:17:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-26T20:17:46.028 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2/2 2024-05-26T20:17:46.118 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2/2 2024-05-26T20:17:46.414 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2/2 2024-05-26T20:17:46.414 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/2 2024-05-26T20:17:46.508 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2/2 2024-05-26T20:17:46.509 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/2 2024-05-26T20:17:46.880 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-26T20:17:46.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:46.880 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:46.881 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-26T20:17:46.881 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:46.881 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:46.998 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-26T20:17:46.998 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:46.998 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:46.998 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-26T20:17:46.999 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:46.999 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:47.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-05-26T20:17:47.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-05-26T20:17:47.861 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:30 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:48.072 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:31 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:49.491 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:49.491 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:49.491 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-05-26T20:17:49.491 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:49.491 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 263 k 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 311 k 2024-05-26T20:17:49.493 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.4 M 2024-05-26T20:17:49.493 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:49.852 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.2.2-1355.gf1882d80.el8 ceph-noarch 263 k 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-05-26T20:17:49.853 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:49.854 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 311 k 2024-05-26T20:17:49.854 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.4 M 2024-05-26T20:17:49.854 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:49.953 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): ceph-volume-18.2.2-1355.gf1882d80.el8.no 645 kB/s | 263 kB 00:00 2024-05-26T20:17:50.100 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): python3-packaging-16.8-9.el8.noarch.rpm 87 kB/s | 48 kB 00:00 2024-05-26T20:17:50.101 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:50.101 INFO:teuthology.orchestra.run.smithi137.stdout:Total 511 kB/s | 311 kB 00:00 2024-05-26T20:17:50.113 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:50.126 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:50.126 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:50.238 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:50.242 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:50.709 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): ceph-volume-18.2.2-1355.gf1882d80.el8.no 397 kB/s | 263 kB 00:00 2024-05-26T20:17:50.950 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): python3-packaging-16.8-9.el8.noarch.rpm 53 kB/s | 48 kB 00:00 2024-05-26T20:17:50.951 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:50.951 INFO:teuthology.orchestra.run.smithi007.stdout:Total 283 kB/s | 311 kB 00:01 2024-05-26T20:17:50.963 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:50.976 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:50.976 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:51.048 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:51.052 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:51.156 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:51.217 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-05-26T20:17:51.675 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-05-26T20:17:51.769 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2/2 2024-05-26T20:17:51.941 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2/2 2024-05-26T20:17:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2/2 2024-05-26T20:17:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 1/2 2024-05-26T20:17:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2/2 2024-05-26T20:17:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 1/2 2024-05-26T20:17:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-05-26T20:17:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:52.571 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-26T20:17:52.572 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:52.572 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:52.813 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-05-26T20:17:52.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:52.813 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:52.813 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.2.2-1355.gf1882d80.el8.noarch 2024-05-26T20:17:52.814 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-26T20:17:52.814 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:52.814 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:52.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-05-26T20:17:53.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-05-26T20:17:53.630 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:36 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:53.837 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:36 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:55.302 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:55.302 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 139 k 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 139 k 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 513 k 2024-05-26T20:17:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:17:55.611 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-18.2.2-1355.gf1882d80.el8.x86_64 450 kB/s | 139 kB 00:00 2024-05-26T20:17:55.612 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:55.612 INFO:teuthology.orchestra.run.smithi137.stdout:Total 448 kB/s | 139 kB 00:00 2024-05-26T20:17:55.612 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:17:55.618 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:17:55.618 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:17:55.626 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 139 k 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:17:55.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:55.628 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 139 k 2024-05-26T20:17:55.628 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 513 k 2024-05-26T20:17:55.628 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:17:55.668 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:17:55.671 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:17:55.784 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:17:55.852 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-18.2.2-1355.gf1882d80.el8.x86_64 620 kB/s | 139 kB 00:00 2024-05-26T20:17:55.853 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:17:55.853 INFO:teuthology.orchestra.run.smithi007.stdout:Total 616 kB/s | 139 kB 00:00 2024-05-26T20:17:55.853 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:17:55.859 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:17:55.859 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:17:55.912 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:17:55.917 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:17:56.033 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:17:56.042 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:56.330 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:57.153 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:17:57.341 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:17:57.577 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:17:57.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:57.577 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:17:57.577 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:17:57.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:17:57.578 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:17:57.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-05-26T20:17:57.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-05-26T20:17:58.345 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:41 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:58.578 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:41 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:17:59.942 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:17:59.989 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:17:59.990 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:17:59.990 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:00.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-05-26T20:18:00.304 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:00.356 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:00.357 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:00.357 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:00.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-05-26T20:18:00.886 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:43 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:01.268 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:44 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:02.623 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 31 k 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:02.624 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:18:02.625 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:02.625 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 k 2024-05-26T20:18:02.625 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 139 k 2024-05-26T20:18:02.625 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:18:02.802 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-18.2.2-1355.gf1882d80.el8.x86_6 175 kB/s | 31 kB 00:00 2024-05-26T20:18:02.803 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:02.803 INFO:teuthology.orchestra.run.smithi137.stdout:Total 173 kB/s | 31 kB 00:00 2024-05-26T20:18:02.803 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:18:02.807 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:18:02.807 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:18:02.835 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:18:02.838 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:18:02.900 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:18:03.044 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 31 k 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2024-05-26T20:18:03.045 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 139 k 2024-05-26T20:18:03.046 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:18:03.141 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:03.197 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-18.2.2-1355.gf1882d80.el8.x86_6 204 kB/s | 31 kB 00:00 2024-05-26T20:18:03.197 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:03.198 INFO:teuthology.orchestra.run.smithi007.stdout:Total 202 kB/s | 31 kB 00:00 2024-05-26T20:18:03.198 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:18:03.202 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:18:03.202 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:18:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:18:03.235 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:18:03.296 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:18:03.391 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:03.480 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:03.771 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:03.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-05-26T20:18:04.136 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:04.136 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:04.137 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:18:04.137 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:04.137 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:04.137 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:04.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-05-26T20:18:04.652 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:47 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:05.047 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:48 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:06.266 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:06.316 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:06.317 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:06.317 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:06.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-05-26T20:18:06.790 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:06.842 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:06.843 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:06.843 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:06.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-05-26T20:18:07.199 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:50 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:07.733 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:50 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:08.820 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:08.869 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:08.870 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:08.870 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:08.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-05-26T20:18:09.457 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:09.511 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:09.513 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:09.513 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:09.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-05-26T20:18:09.753 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:52 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:53 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:11.377 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:11.426 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:11.427 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:11.427 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:11.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-05-26T20:18:12.161 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:12.216 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:12.217 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:12.217 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:12.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-05-26T20:18:12.323 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:55 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:13.116 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:56 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:13.928 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:13.977 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:13.978 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:13.978 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:14.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-05-26T20:18:14.841 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:14.850 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:57 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:14.895 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:14.895 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:15.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-05-26T20:18:15.800 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:58 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:16.462 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:16.511 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:16.512 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:16.513 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:16.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-05-26T20:18:17.389 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:03:00 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:17.584 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:17.585 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:17.585 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:17.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-05-26T20:18:18.476 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:01 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:19.002 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:19.053 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:19.054 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-05-26T20:18:19.054 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:19.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-05-26T20:18:19.937 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:03:02 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:20.202 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:18.2.2-1355.gf1882d80.el8.x86_64 is already installed. 2024-05-26T20:18:20.254 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:20.255 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-05-26T20:18:20.256 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:20.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-05-26T20:18:21.139 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:04 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:21.606 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 95 k 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:21.607 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 95 k 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 280 k 2024-05-26T20:18:21.608 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:18:21.820 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-18.2.2-1355.gf1882d80.el8.x86_64.rpm 449 kB/s | 95 kB 00:00 2024-05-26T20:18:21.820 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:21.820 INFO:teuthology.orchestra.run.smithi137.stdout:Total 446 kB/s | 95 kB 00:00 2024-05-26T20:18:21.821 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:18:21.831 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:18:21.831 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:18:21.880 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:18:21.883 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:18:21.996 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:18:22.246 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:22.918 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:22.918 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:22.919 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-05-26T20:18:22.919 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:22.919 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:18:22.919 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 95 k 2024-05-26T20:18:22.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:22.921 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:18:22.921 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:22.921 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:18:22.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:22.922 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 95 k 2024-05-26T20:18:22.922 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 280 k 2024-05-26T20:18:22.922 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:18:22.943 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:22.944 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:22.944 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:18:22.944 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:22.944 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:22.944 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:23.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-05-26T20:18:23.409 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-18.2.2-1355.gf1882d80.el8.x86_64.rpm 194 kB/s | 95 kB 00:00 2024-05-26T20:18:23.410 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:23.410 INFO:teuthology.orchestra.run.smithi007.stdout:Total 193 kB/s | 95 kB 00:00 2024-05-26T20:18:23.411 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:18:23.422 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:18:23.423 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:18:23.478 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:18:23.482 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:18:23.630 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:18:23.810 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:23.980 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:03:06 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:24.106 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:24.507 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:24.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-05-26T20:18:25.489 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:08 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:25.646 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.5 M 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:18:25.647 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:25.648 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.5 M 2024-05-26T20:18:25.648 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 16 M 2024-05-26T20:18:25.648 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:18:26.775 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-18.2.2-1355.gf1882d80.el8.x86_64.rpm 3.1 MB/s | 3.5 MB 00:01 2024-05-26T20:18:26.776 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:26.776 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.1 MB/s | 3.5 MB 00:01 2024-05-26T20:18:26.777 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:18:26.789 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:18:26.789 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:18:26.850 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:18:26.853 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:18:27.274 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:27.274 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:27.274 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-05-26T20:18:27.274 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:27.274 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 3.5 M 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:18:27.275 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:27.276 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.5 M 2024-05-26T20:18:27.276 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 16 M 2024-05-26T20:18:27.276 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:18:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:18:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:27.892 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:28.279 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:28.279 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:28.279 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:18:28.279 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:28.280 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:28.280 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:28.305 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-18.2.2-1355.gf1882d80.el8.x86_64.rpm 3.4 MB/s | 3.5 MB 00:01 2024-05-26T20:18:28.306 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:28.306 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.3 MB/s | 3.5 MB 00:01 2024-05-26T20:18:28.307 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:18:28.322 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:18:28.322 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:18:28.393 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:18:28.396 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:18:28.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-05-26T20:18:28.862 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:18:29.119 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:29.362 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:03:12 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:29.999 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:29.999 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:29.999 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:18:30.000 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:30.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:30.000 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:30.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-05-26T20:18:31.022 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-05-26T20:18:31.023 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-05-26T20:18:31.023 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:31.023 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 182 k 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:31.024 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 182 k 2024-05-26T20:18:31.025 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 572 k 2024-05-26T20:18:31.025 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-05-26T20:18:31.068 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:14 ago on Sun 26 May 2024 08:15:17 PM UTC. 2024-05-26T20:18:31.312 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-18.2.2-1355.gf1882d80.el8.x86_64.rpm 633 kB/s | 182 kB 00:00 2024-05-26T20:18:31.313 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:31.313 INFO:teuthology.orchestra.run.smithi137.stdout:Total 630 kB/s | 182 kB 00:00 2024-05-26T20:18:31.313 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-05-26T20:18:31.324 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-05-26T20:18:31.324 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-05-26T20:18:31.374 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-05-26T20:18:31.377 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-05-26T20:18:31.497 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-05-26T20:18:31.736 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:32.025 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:32.442 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-05-26T20:18:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:32.443 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-05-26T20:18:32.726 DEBUG:teuthology.parallel:result is None 2024-05-26T20:18:32.862 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-05-26T20:18:32.862 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.2.2-1355.gf1882d80.el8 ceph 182 k 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-05-26T20:18:32.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:32.864 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 182 k 2024-05-26T20:18:32.864 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 572 k 2024-05-26T20:18:32.864 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-05-26T20:18:33.168 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-18.2.2-1355.gf1882d80.el8.x86_64.rpm 599 kB/s | 182 kB 00:00 2024-05-26T20:18:33.168 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-05-26T20:18:33.168 INFO:teuthology.orchestra.run.smithi007.stdout:Total 597 kB/s | 182 kB 00:00 2024-05-26T20:18:33.169 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-05-26T20:18:33.181 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-05-26T20:18:33.182 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-05-26T20:18:33.236 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-05-26T20:18:33.239 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-05-26T20:18:33.357 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-05-26T20:18:33.542 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:33.840 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:34.225 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 1/1 2024-05-26T20:18:34.225 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:34.226 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-05-26T20:18:34.226 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.2.2-1355.gf1882d80.el8.x86_64 2024-05-26T20:18:34.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:34.226 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-05-26T20:18:34.465 DEBUG:teuthology.parallel:result is None 2024-05-26T20:18:34.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=f1882d8093fb4b60c1d1e52444565787ca4bd6cc 2024-05-26T20:18:34.606 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-26T20:18:34.630 INFO:teuthology.orchestra.run.smithi007.stdout:18.2.2-1355.gf1882d80.el8 2024-05-26T20:18:34.630 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1355.gf1882d80.el8 2024-05-26T20:18:34.630 INFO:teuthology.task.install:The correct ceph version 18.2.2-1355.gf1882d80 is installed. 2024-05-26T20:18:34.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=f1882d8093fb4b60c1d1e52444565787ca4bd6cc 2024-05-26T20:18:34.853 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-26T20:18:34.880 INFO:teuthology.orchestra.run.smithi137.stdout:18.2.2-1355.gf1882d80.el8 2024-05-26T20:18:34.880 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1355.gf1882d80.el8 2024-05-26T20:18:34.881 INFO:teuthology.task.install:The correct ceph version 18.2.2-1355.gf1882d80 is installed. 2024-05-26T20:18:34.882 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-26T20:18:34.883 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:34.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-26T20:18:34.912 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:34.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-26T20:18:34.952 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-26T20:18:34.952 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:34.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-05-26T20:18:34.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-26T20:18:35.049 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:35.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-05-26T20:18:35.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-26T20:18:35.150 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-26T20:18:35.150 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:35.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-26T20:18:35.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-26T20:18:35.244 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:35.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-26T20:18:35.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-26T20:18:35.342 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-26T20:18:35.343 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:35.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-05-26T20:18:35.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-26T20:18:35.436 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:35.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-05-26T20:18:35.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-26T20:18:35.533 INFO:teuthology.run_tasks:Running task ceph... 2024-05-26T20:18:35.608 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-26T20:18:35.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-05-26T20:18:35.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-05-26T20:18:35.639 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-26T20:18:35.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-26T20:18:35.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-26T20:18:35.706 INFO:tasks.ceph:Creating extra log directories... 2024-05-26T20:18:35.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-26T20:18:35.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-26T20:18:35.785 INFO:tasks.ceph:Creating ceph cluster c1... 2024-05-26T20:18:35.786 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-26T20:18:35.786 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7725509', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1296, 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBCfhtYy3Qw3mwMAX9p72Gz0S7a+LiV/khZYrpw0xSead6z7IlHJ4uA9wVBZvncokQwdr9xZoMKI5ZkZ3+i33c=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFdhq4LVhtbDhGEnFztzQHEDXgbHB7CDfQGa/mAGD0p7mZ1B2lRDq+A7AtI9SE0/anQeadF/vf9q0uiaPCFTsQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e7ae7580c417d6ab27fda98fecadca5d9f85d63e', 'timestamp': '2024-05-24_22:40:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-26T20:18:35.786 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-05-26T20:18:35.814 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-05-26T20:18:35.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-05-26T20:18:35.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-05-26T20:18:35.898 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:35.899 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-05-26T20:18:35.955 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-26T20:18:35.955 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 170529 Links: 1 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-05-26 20:18:29.304844005 +0000 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-05-26 20:14:03.773439561 +0000 2024-05-26T20:18:36.013 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-05-26 20:14:03.773439561 +0000 2024-05-26T20:18:36.014 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-05-26T20:18:36.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-26T20:18:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-05-26T20:18:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-05-26T20:18:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.4645e-05 s, 5.4 MB/s 2024-05-26T20:18:36.081 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-26T20:18:36.140 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-05-26T20:18:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-26T20:18:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:18:36.197 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 174401 Links: 1 2024-05-26T20:18:36.197 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:18:36.198 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:18:36.198 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-05-26 20:18:29.304844005 +0000 2024-05-26T20:18:36.198 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-05-26 20:14:04.116434439 +0000 2024-05-26T20:18:36.198 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-05-26 20:14:04.116434439 +0000 2024-05-26T20:18:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-05-26T20:18:36.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-26T20:18:36.263 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-05-26T20:18:36.263 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-05-26T20:18:36.263 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.7986e-05 s, 5.2 MB/s 2024-05-26T20:18:36.264 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-26T20:18:36.324 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 175490 Links: 1 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-05-26 20:18:29.304844005 +0000 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-05-26 20:14:04.557427854 +0000 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-05-26 20:14:04.557427854 +0000 2024-05-26T20:18:36.382 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-05-26T20:18:36.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-26T20:18:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-05-26T20:18:36.448 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-05-26T20:18:36.448 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.9516e-05 s, 5.7 MB/s 2024-05-26T20:18:36.448 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-26T20:18:36.507 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 170767 Links: 1 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-05-26 20:18:29.304844005 +0000 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-05-26 20:14:04.897422777 +0000 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-05-26 20:14:04.897422777 +0000 2024-05-26T20:18:36.566 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-05-26T20:18:36.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-26T20:18:36.631 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-05-26T20:18:36.631 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-05-26T20:18:36.631 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.831e-05 s, 5.2 MB/s 2024-05-26T20:18:36.632 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-26T20:18:36.691 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:18:36.691 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-26T20:18:36.691 INFO:tasks.ceph:Generating config... 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] debug rgw = 20 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-05-26T20:18:36.693 INFO:tasks.ceph:[client] setgroup = ceph 2024-05-26T20:18:36.694 INFO:tasks.ceph:[client] setuser = ceph 2024-05-26T20:18:36.694 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-05-26T20:18:36.694 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-26T20:18:36.694 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-26T20:18:36.694 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-26T20:18:36.694 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-26T20:18:36.694 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-05-26T20:18:36.694 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-05-26T20:18:36.695 INFO:tasks.ceph:Setting up c1.mon.a... 2024-05-26T20:18:36.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-05-26T20:18:36.791 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/c1.keyring 2024-05-26T20:18:36.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-05-26T20:18:36.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-05-26T20:18:36.866 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.7')] 2024-05-26T20:18:36.867 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-05-26T20:18:36.868 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:36.868 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-26T20:18:36.925 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --print /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:37.007 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:18:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:18:37.008 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:37.008 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 7f28eddc-9cea-4060-87c7-2f02a73c380b 2024-05-26T20:18:37.008 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2024-05-26T20:18:37.008 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 7f28eddc-9cea-4060-87c7-2f02a73c380b 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-05-26T20:18:37.009341+0000 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-05-26T20:18:37.009341+0000 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-05-26T20:18:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-05-26T20:18:37.010 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-26T20:18:37.067 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 7f28eddc-9cea-4060-87c7-2f02a73c380b... 2024-05-26T20:18:37.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-05-26T20:18:37.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.153 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-05-26T20:18:37.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-05-26T20:18:37.155 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-05-26T20:18:37.156 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.7 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 7f28eddc-9cea-4060-87c7-2f02a73c380b 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-05-26T20:18:37.157 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-05-26T20:18:37.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: rocksdb delete range threshold = 0 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-05-26T20:18:37.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-05-26T20:18:37.160 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-05-26T20:18:37.161 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw = 20 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw sync = 20 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt require ssl = False 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt s3 kms backend = testing 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw curl low speed time = 300 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw data log num shards = 4 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw md log max shards = 4 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync data inject err probability = 0.1 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync log trim interval = 0 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync meta inject err probability = 0.1 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync obj etag verify = True 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: setgroup = ceph 2024-05-26T20:18:37.162 INFO:teuthology.orchestra.run.smithi007.stdout: setuser = ceph 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout: rgw data notify interval msec = 0 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 10 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 1000 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 10000 2024-05-26T20:18:37.163 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-05-26T20:18:37.165 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-05-26T20:18:37.166 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.7 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 7f28eddc-9cea-4060-87c7-2f02a73c380b 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-05-26T20:18:37.167 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-05-26T20:18:37.168 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-05-26T20:18:37.169 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-05-26T20:18:37.170 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-05-26T20:18:37.171 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw sync = 20 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync data inject err probability = 0.1 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2024-05-26T20:18:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync meta inject err probability = 0.1 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync obj etag verify = True 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data notify interval msec = 0 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-05-26T20:18:37.173 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2024-05-26T20:18:37.174 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-05-26T20:18:37.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-05-26T20:18:37.273 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-26T20:18:37.273 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:37.273 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-05-26T20:18:37.291 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:37.291 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-05-26T20:18:37.348 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:18:37.349 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:37.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/c1.keyring 2024-05-26T20:18:37.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-05-26T20:18:37.426 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:37.426 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:37.484 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:18:37.484 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:37.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/c1.keyring 2024-05-26T20:18:37.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-05-26T20:18:37.527 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:18:37.527 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:37.587 INFO:tasks.ceph:Setting up mon nodes... 2024-05-26T20:18:37.587 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-26T20:18:37.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-05-26T20:18:37.642 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-05-26T20:18:37.645 INFO:tasks.ceph:Setting up mds nodes... 2024-05-26T20:18:37.645 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-26T20:18:37.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-05-26T20:18:37.689 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/c1.client.0.keyring 2024-05-26T20:18:37.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-05-26T20:18:37.746 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/c1.client.1.keyring 2024-05-26T20:18:37.758 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-26T20:18:37.758 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-26T20:18:37.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-05-26T20:18:37.785 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:18:37.785 INFO:tasks.ceph:role: c1.osd.0 2024-05-26T20:18:37.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:18:37.786 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-26T20:18:37.853 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:18:37.853 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:18:37.853 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:18:37.854 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:18:37.855 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:18:37.875 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-05-26T20:18:37.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-05-26T20:18:37.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-05-26T20:18:37.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-05-26T20:18:38.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-05-26T20:18:38.099 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:18:38.099 INFO:tasks.ceph:role: c1.osd.1 2024-05-26T20:18:38.100 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:18:38.100 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:18:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:18:38.189 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-05-26T20:18:38.191 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-05-26T20:18:38.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-05-26T20:18:38.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-05-26T20:18:38.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-05-26T20:18:38.411 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:18:38.411 INFO:tasks.ceph:role: c1.osd.2 2024-05-26T20:18:38.411 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:18:38.411 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-26T20:18:38.479 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:18:38.479 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:18:38.479 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:18:38.479 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:18:38.479 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:18:38.480 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:18:38.480 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:18:38.480 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:18:38.480 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:18:38.480 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:18:38.501 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-05-26T20:18:38.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-05-26T20:18:38.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-05-26T20:18:38.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-05-26T20:18:38.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:38.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.751+0000 7f6e7eb2f7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-05-26T20:18:38.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.751+0000 7f6e7eb2f7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-05-26T20:18:38.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.751+0000 7f6e7eb2f7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-05-26T20:18:38.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.751+0000 7f6e7eb2f7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-05-26T20:18:38.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.751+0000 7f6e7eb2f7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-05-26T20:18:38.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:38.752+0000 7f6e7eb2f7c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-05-26T20:18:40.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-05-26T20:18:40.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.959+0000 7f850bcef7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.960+0000 7f850bcef7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.960+0000 7f850bcef7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.960+0000 7f850bcef7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.960+0000 7f850bcef7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-05-26T20:18:40.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:40.960+0000 7f850bcef7c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-05-26T20:18:43.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-05-26T20:18:43.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:43.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.169+0000 7f488a12b7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-05-26T20:18:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.169+0000 7f488a12b7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-05-26T20:18:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.169+0000 7f488a12b7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-05-26T20:18:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.169+0000 7f488a12b7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-05-26T20:18:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.169+0000 7f488a12b7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-05-26T20:18:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:18:43.170+0000 7f488a12b7c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-05-26T20:18:45.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-05-26T20:18:45.280 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-26T20:18:45.280 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-05-26T20:18:45.351 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-05-26T20:18:45.423 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-05-26T20:18:45.493 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-05-26T20:18:45.563 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.563 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-05-26T20:18:45.623 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:45.623 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-05-26T20:18:45.682 INFO:tasks.ceph:Adding keys to all mons... 2024-05-26T20:18:45.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/c1.keyring 2024-05-26T20:18:45.749 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAdmVNmPBdeJhAAH2b18LrTabjM7eqrATe39Q== 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAemVNm74/cLBAAj0rJmHnfE8bQtZlnzj1bLQ== 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAgmVNmuwNHORAAGMp0DBv4x/I9mW/4C3ioUg== 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAjmVNm28ApChAAnikyeZ9yGkoBDmyQBR9sFg== 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAdmVNmeTwvKRAANKvucygZ2XXnOIa7TpIPYQ== 2024-05-26T20:18:45.750 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2024-05-26T20:18:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAdmVNmD42PLBAAdBGNcQKPJs4aSY/9lsYifA== 2024-05-26T20:18:45.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-26T20:18:45.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:18:45.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:18:45.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:18:46.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-26T20:18:46.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-26T20:18:46.128 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-26T20:18:46.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-05-26T20:18:46.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-05-26T20:18:46.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-05-26T20:18:46.282 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-05-26T20:18:46.339 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-05-26T20:18:46.340 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-05-26T20:18:46.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-05-26T20:18:46.382 INFO:tasks.ceph.c1.mon.a:Started 2024-05-26T20:18:46.382 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-05-26T20:18:46.382 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-05-26T20:18:46.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-05-26T20:18:46.384 INFO:tasks.ceph.c1.mgr.x:Started 2024-05-26T20:18:46.384 INFO:tasks.ceph:Setting crush tunables to default 2024-05-26T20:18:46.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd crush tunables default 2024-05-26T20:18:46.595 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-05-26T20:18:46.606 INFO:tasks.ceph:check_enable_crimson: False 2024-05-26T20:18:46.606 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-05-26T20:18:46.606 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:46.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-05-26T20:18:46.633 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:46.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-05-26T20:18:46.702 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:18:46.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-05-26T20:18:46.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd new 25101e94-2e1d-4957-9058-312cf8d7106f 0 2024-05-26T20:18:46.792 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:46.792+0000 7ffbdcd30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T20:18:46.990 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:46.990+0000 7ffbdcd30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T20:18:47.073 INFO:teuthology.orchestra.run.smithi007.stdout:0 2024-05-26T20:18:47.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd new ebb0220c-2f10-4fec-8973-642764113c92 1 2024-05-26T20:18:47.307 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-05-26T20:18:47.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd new 28a67f50-717f-4815-921d-a7ae3de5f4bc 2 2024-05-26T20:18:47.404 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:47.404+0000 7ffbdcd30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T20:18:47.554 INFO:teuthology.orchestra.run.smithi007.stdout:2 2024-05-26T20:18:47.567 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-05-26T20:18:47.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-05-26T20:18:47.569 INFO:tasks.ceph.c1.osd.0:Started 2024-05-26T20:18:47.569 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-05-26T20:18:47.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-05-26T20:18:47.571 INFO:tasks.ceph.c1.osd.1:Started 2024-05-26T20:18:47.571 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-05-26T20:18:47.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-05-26T20:18:47.573 INFO:tasks.ceph.c1.osd.2:Started 2024-05-26T20:18:47.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-05-26T20:18:47.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:47.810 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"fsid":"7f28eddc-9cea-4060-87c7-2f02a73c380b","created":"2024-05-26T20:18:46.490538+0000","modified":"2024-05-26T20:18:47.554130+0000","last_up_change":"0.000000","last_in_change":"2024-05-26T20:18:47.554130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"25101e94-2e1d-4957-9058-312cf8d7106f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ebb0220c-2f10-4fec-8973-642764113c92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28a67f50-717f-4815-921d-a7ae3de5f4bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:18:47.823 INFO:tasks.ceph.ceph_manager.c1:[] 2024-05-26T20:18:47.823 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-26T20:18:47.998 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:47.998+0000 7ffbdcd30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T20:18:48.125 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-05-26T20:18:48.175 INFO:tasks.ceph.c1.osd.1.smithi007.stderr:2024-05-26T20:18:48.175+0000 7f33c54987c0 -1 Falling back to public interface 2024-05-26T20:18:48.180 INFO:tasks.ceph.c1.osd.0.smithi007.stderr:2024-05-26T20:18:48.180+0000 7f9742be07c0 -1 Falling back to public interface 2024-05-26T20:18:48.199 INFO:tasks.ceph.c1.osd.2.smithi007.stderr:2024-05-26T20:18:48.199+0000 7f9ed209c7c0 -1 Falling back to public interface 2024-05-26T20:18:48.222 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:48.222+0000 7ffbdcd30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T20:18:48.324 INFO:teuthology.misc.health.smithi007.stdout: 2024-05-26T20:18:48.324 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":5,"fsid":"7f28eddc-9cea-4060-87c7-2f02a73c380b","created":"2024-05-26T20:18:46.490538+0000","modified":"2024-05-26T20:18:47.554130+0000","last_up_change":"0.000000","last_in_change":"2024-05-26T20:18:47.554130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"25101e94-2e1d-4957-9058-312cf8d7106f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ebb0220c-2f10-4fec-8973-642764113c92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28a67f50-717f-4815-921d-a7ae3de5f4bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:18:48.335 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-26T20:18:48.615 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:48.615+0000 7ffbdcd30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T20:18:49.319 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.319+0000 7ffbdcd30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T20:18:49.395 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.395+0000 7ffbdcd30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T20:18:49.537 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.537+0000 7ffbdcd30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T20:18:49.717 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.717+0000 7ffbdcd30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T20:18:49.794 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.794+0000 7ffbdcd30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T20:18:49.877 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.877+0000 7ffbdcd30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T20:18:49.957 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:49.957+0000 7ffbdcd30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T20:18:50.047 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:50.047+0000 7ffbdcd30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T20:18:50.260 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:50.260+0000 7ffbdcd30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T20:18:50.481 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:50.481+0000 7ffbdcd30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T20:18:50.653 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:50.653+0000 7ffbdcd30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T20:18:50.728 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:50.728+0000 7ffbdcd30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T20:18:50.896 INFO:tasks.ceph.c1.osd.1.smithi007.stderr:2024-05-26T20:18:50.896+0000 7f33c54987c0 -1 osd.1 0 log_to_monitors true 2024-05-26T20:18:50.919 INFO:tasks.ceph.c1.osd.0.smithi007.stderr:2024-05-26T20:18:50.919+0000 7f9742be07c0 -1 osd.0 0 log_to_monitors true 2024-05-26T20:18:50.941 INFO:tasks.ceph.c1.osd.2.smithi007.stderr:2024-05-26T20:18:50.941+0000 7f9ed209c7c0 -1 osd.2 0 log_to_monitors true 2024-05-26T20:18:51.452 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:51.453+0000 7ffbdcd30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T20:18:51.625 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:51.625+0000 7ffbdcd30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T20:18:51.792 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:51.792+0000 7ffbdcd30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T20:18:52.263 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:52.263+0000 7ffbdcd30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T20:18:52.460 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:52.460+0000 7ffbdcd30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T20:18:52.689 INFO:tasks.ceph.c1.osd.0.smithi007.stderr:2024-05-26T20:18:52.689+0000 7f9736f82700 -1 osd.0 0 waiting for initial osdmap 2024-05-26T20:18:52.694 INFO:tasks.ceph.c1.osd.1.smithi007.stderr:2024-05-26T20:18:52.694+0000 7f33b983a700 -1 osd.1 0 waiting for initial osdmap 2024-05-26T20:18:52.700 INFO:tasks.ceph.c1.osd.0.smithi007.stderr:2024-05-26T20:18:52.700+0000 7f972eb5c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:18:52.702 INFO:tasks.ceph.c1.osd.1.smithi007.stderr:2024-05-26T20:18:52.703+0000 7f33b1414700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:18:52.716 INFO:tasks.ceph.c1.osd.2.smithi007.stderr:2024-05-26T20:18:52.716+0000 7f9ec643e700 -1 osd.2 0 waiting for initial osdmap 2024-05-26T20:18:52.732 INFO:tasks.ceph.c1.osd.2.smithi007.stderr:2024-05-26T20:18:52.732+0000 7f9ebe018700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:18:52.875 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:52.875+0000 7ffbdcd30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T20:18:52.980 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:52.980+0000 7ffbdcd30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T20:18:53.990 INFO:tasks.ceph.c1.mgr.x.smithi007.stderr:2024-05-26T20:18:53.990+0000 7ffb9e11a700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-26T20:18:54.638 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-05-26T20:18:54.982 INFO:teuthology.misc.health.smithi007.stdout: 2024-05-26T20:18:54.982 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":9,"fsid":"7f28eddc-9cea-4060-87c7-2f02a73c380b","created":"2024-05-26T20:18:46.490538+0000","modified":"2024-05-26T20:18:54.697310+0000","last_up_change":"2024-05-26T20:18:53.696499+0000","last_in_change":"2024-05-26T20:18:47.554130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"25101e94-2e1d-4957-9058-312cf8d7106f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":41964},{"type":"v1","addr":"172.21.15.7:6809","nonce":41964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":41964},{"type":"v1","addr":"172.21.15.7:6811","nonce":41964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":41964},{"type":"v1","addr":"172.21.15.7:6815","nonce":41964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":41964},{"type":"v1","addr":"172.21.15.7:6813","nonce":41964}]},"public_addr":"172.21.15.7:6809/41964","cluster_addr":"172.21.15.7:6811/41964","heartbeat_back_addr":"172.21.15.7:6815/41964","heartbeat_front_addr":"172.21.15.7:6813/41964","state":["exists","up"]},{"osd":1,"uuid":"ebb0220c-2f10-4fec-8973-642764113c92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":41965},{"type":"v1","addr":"172.21.15.7:6801","nonce":41965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":41965},{"type":"v1","addr":"172.21.15.7:6803","nonce":41965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":41965},{"type":"v1","addr":"172.21.15.7:6807","nonce":41965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":41965},{"type":"v1","addr":"172.21.15.7:6805","nonce":41965}]},"public_addr":"172.21.15.7:6801/41965","cluster_addr":"172.21.15.7:6803/41965","heartbeat_back_addr":"172.21.15.7:6807/41965","heartbeat_front_addr":"172.21.15.7:6805/41965","state":["exists","up"]},{"osd":2,"uuid":"28a67f50-717f-4815-921d-a7ae3de5f4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":41986},{"type":"v1","addr":"172.21.15.7:6817","nonce":41986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":41986},{"type":"v1","addr":"172.21.15.7:6819","nonce":41986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":41986},{"type":"v1","addr":"172.21.15.7:6823","nonce":41986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":41986},{"type":"v1","addr":"172.21.15.7:6821","nonce":41986}]},"public_addr":"172.21.15.7:6817/41986","cluster_addr":"172.21.15.7:6819/41986","heartbeat_back_addr":"172.21.15.7:6823/41986","heartbeat_front_addr":"172.21.15.7:6821/41986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.881809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.907840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:52.717616+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:18:54.995 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-26T20:18:54.995 INFO:tasks.ceph:Creating RBD pool 2024-05-26T20:18:54.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd pool create rbd 8 2024-05-26T20:18:55.998 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-05-26T20:18:56.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-26T20:18:57.003 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-26T20:18:57.019 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-05-26T20:18:57.020 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-05-26T20:18:57.020 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-05-26T20:18:57.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-05-26T20:18:57.020 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-26T20:18:57.482 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:57.497 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":41552},{"type":"v1","addr":"172.21.15.7:6825","nonce":41552}]},"active_addr":"172.21.15.7:6825/41552","active_change":"2024-05-26T20:18:52.982832+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2467563136}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":402359315}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2487858074}]}]} 2024-05-26T20:18:57.499 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-05-26T20:18:57.499 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-05-26T20:18:57.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-05-26T20:18:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:57.835 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"fsid":"7f28eddc-9cea-4060-87c7-2f02a73c380b","created":"2024-05-26T20:18:46.490538+0000","modified":"2024-05-26T20:18:56.990635+0000","last_up_change":"2024-05-26T20:18:53.696499+0000","last_in_change":"2024-05-26T20:18:47.554130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T20:18:55.026897+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-26T20:18:55.361971+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25101e94-2e1d-4957-9058-312cf8d7106f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":41964},{"type":"v1","addr":"172.21.15.7:6809","nonce":41964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":41964},{"type":"v1","addr":"172.21.15.7:6811","nonce":41964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":41964},{"type":"v1","addr":"172.21.15.7:6815","nonce":41964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":41964},{"type":"v1","addr":"172.21.15.7:6813","nonce":41964}]},"public_addr":"172.21.15.7:6809/41964","cluster_addr":"172.21.15.7:6811/41964","heartbeat_back_addr":"172.21.15.7:6815/41964","heartbeat_front_addr":"172.21.15.7:6813/41964","state":["exists","up"]},{"osd":1,"uuid":"ebb0220c-2f10-4fec-8973-642764113c92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":41965},{"type":"v1","addr":"172.21.15.7:6801","nonce":41965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":41965},{"type":"v1","addr":"172.21.15.7:6803","nonce":41965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":41965},{"type":"v1","addr":"172.21.15.7:6807","nonce":41965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":41965},{"type":"v1","addr":"172.21.15.7:6805","nonce":41965}]},"public_addr":"172.21.15.7:6801/41965","cluster_addr":"172.21.15.7:6803/41965","heartbeat_back_addr":"172.21.15.7:6807/41965","heartbeat_front_addr":"172.21.15.7:6805/41965","state":["exists","up"]},{"osd":2,"uuid":"28a67f50-717f-4815-921d-a7ae3de5f4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":41986},{"type":"v1","addr":"172.21.15.7:6817","nonce":41986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":41986},{"type":"v1","addr":"172.21.15.7:6819","nonce":41986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":41986},{"type":"v1","addr":"172.21.15.7:6823","nonce":41986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":41986},{"type":"v1","addr":"172.21.15.7:6821","nonce":41986}]},"public_addr":"172.21.15.7:6817/41986","cluster_addr":"172.21.15.7:6819/41986","heartbeat_back_addr":"172.21.15.7:6823/41986","heartbeat_front_addr":"172.21.15.7:6821/41986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.881809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.907840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:52.717616+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:18:57.849 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-05-26T20:18:57.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-05-26T20:18:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:18:58.187 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"fsid":"7f28eddc-9cea-4060-87c7-2f02a73c380b","created":"2024-05-26T20:18:46.490538+0000","modified":"2024-05-26T20:18:57.993300+0000","last_up_change":"2024-05-26T20:18:53.696499+0000","last_in_change":"2024-05-26T20:18:47.554130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T20:18:55.026897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-26T20:18:55.361971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25101e94-2e1d-4957-9058-312cf8d7106f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":41964},{"type":"v1","addr":"172.21.15.7:6809","nonce":41964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":41964},{"type":"v1","addr":"172.21.15.7:6811","nonce":41964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":41964},{"type":"v1","addr":"172.21.15.7:6815","nonce":41964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":41964},{"type":"v1","addr":"172.21.15.7:6813","nonce":41964}]},"public_addr":"172.21.15.7:6809/41964","cluster_addr":"172.21.15.7:6811/41964","heartbeat_back_addr":"172.21.15.7:6815/41964","heartbeat_front_addr":"172.21.15.7:6813/41964","state":["exists","up"]},{"osd":1,"uuid":"ebb0220c-2f10-4fec-8973-642764113c92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":41965},{"type":"v1","addr":"172.21.15.7:6801","nonce":41965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":41965},{"type":"v1","addr":"172.21.15.7:6803","nonce":41965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":41965},{"type":"v1","addr":"172.21.15.7:6807","nonce":41965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":41965},{"type":"v1","addr":"172.21.15.7:6805","nonce":41965}]},"public_addr":"172.21.15.7:6801/41965","cluster_addr":"172.21.15.7:6803/41965","heartbeat_back_addr":"172.21.15.7:6807/41965","heartbeat_front_addr":"172.21.15.7:6805/41965","state":["exists","up"]},{"osd":2,"uuid":"28a67f50-717f-4815-921d-a7ae3de5f4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":41986},{"type":"v1","addr":"172.21.15.7:6817","nonce":41986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":41986},{"type":"v1","addr":"172.21.15.7:6819","nonce":41986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":41986},{"type":"v1","addr":"172.21.15.7:6823","nonce":41986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":41986},{"type":"v1","addr":"172.21.15.7:6821","nonce":41986}]},"public_addr":"172.21.15.7:6817/41986","cluster_addr":"172.21.15.7:6819/41986","heartbeat_back_addr":"172.21.15.7:6823/41986","heartbeat_front_addr":"172.21.15.7:6821/41986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.881809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:51.907840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:18:52.717616+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:18:58.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-05-26T20:18:58.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-05-26T20:18:58.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-05-26T20:18:58.370 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:18:58.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-05-26T20:18:58.375 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:18:58.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-05-26T20:18:58.377 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:18:58.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-05-26T20:18:58.721 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-05-26T20:18:58.736 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2024-05-26T20:18:58.774 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-05-26T20:18:58.779 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-05-26T20:18:58.792 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2024-05-26T20:18:58.796 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-05-26T20:18:59.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-05-26T20:18:59.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-05-26T20:18:59.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-05-26T20:19:00.113 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:19:00.134 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2024-05-26T20:19:00.134 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:00.160 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:19:00.174 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2024-05-26T20:19:00.174 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:00.205 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-05-26T20:19:00.219 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-05-26T20:19:00.219 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:00.220 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-05-26T20:19:00.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-05-26T20:19:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-05-26T20:19:00.612 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-05-26T20:18:58.987708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82880,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489920,"statfs":{"total":322122547200,"available":322037678080,"internally_reserved":0,"allocated":1482752,"data_stored":1255679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.003062+0000","last_change":"2024-05-26T20:18:57.002225+0000","last_active":"2024-05-26T20:18:57.003062+0000","last_peered":"2024-05-26T20:18:57.003062+0000","last_clean":"2024-05-26T20:18:57.003062+0000","last_became_active":"2024-05-26T20:18:57.001923+0000","last_became_peered":"2024-05-26T20:18:57.001923+0000","last_unstale":"2024-05-26T20:18:57.003062+0000","last_undegraded":"2024-05-26T20:18:57.003062+0000","last_fullsized":"2024-05-26T20:18:57.003062+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:05:43.587224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.001014+0000","last_change":"2024-05-26T20:18:56.999962+0000","last_active":"2024-05-26T20:18:57.001014+0000","last_peered":"2024-05-26T20:18:57.001014+0000","last_clean":"2024-05-26T20:18:57.001014+0000","last_became_active":"2024-05-26T20:18:56.999651+0000","last_became_peered":"2024-05-26T20:18:56.999651+0000","last_unstale":"2024-05-26T20:18:57.001014+0000","last_undegraded":"2024-05-26T20:18:57.001014+0000","last_fullsized":"2024-05-26T20:18:57.001014+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:38.570740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.002033+0000","last_change":"2024-05-26T20:18:57.001156+0000","last_active":"2024-05-26T20:18:57.002033+0000","last_peered":"2024-05-26T20:18:57.002033+0000","last_clean":"2024-05-26T20:18:57.002033+0000","last_became_active":"2024-05-26T20:18:57.000872+0000","last_became_peered":"2024-05-26T20:18:57.000872+0000","last_unstale":"2024-05-26T20:18:57.002033+0000","last_undegraded":"2024-05-26T20:18:57.002033+0000","last_fullsized":"2024-05-26T20:18:57.002033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:25:26.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.997080+0000","last_change":"2024-05-26T20:18:57.002192+0000","last_active":"2024-05-26T20:18:57.997080+0000","last_peered":"2024-05-26T20:18:57.997080+0000","last_clean":"2024-05-26T20:18:57.997080+0000","last_became_active":"2024-05-26T20:18:57.001799+0000","last_became_peered":"2024-05-26T20:18:57.001799+0000","last_unstale":"2024-05-26T20:18:57.997080+0000","last_undegraded":"2024-05-26T20:18:57.997080+0000","last_fullsized":"2024-05-26T20:18:57.997080+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:41:21.946315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.492994+0000","last_change":"2024-05-26T20:18:57.003988+0000","last_active":"2024-05-26T20:18:57.492994+0000","last_peered":"2024-05-26T20:18:57.492994+0000","last_clean":"2024-05-26T20:18:57.492994+0000","last_became_active":"2024-05-26T20:18:57.003697+0000","last_became_peered":"2024-05-26T20:18:57.003697+0000","last_unstale":"2024-05-26T20:18:57.492994+0000","last_undegraded":"2024-05-26T20:18:57.492994+0000","last_fullsized":"2024-05-26T20:18:57.492994+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:46:27.924057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.001075+0000","last_change":"2024-05-26T20:18:56.999481+0000","last_active":"2024-05-26T20:18:57.001075+0000","last_peered":"2024-05-26T20:18:57.001075+0000","last_clean":"2024-05-26T20:18:57.001075+0000","last_became_active":"2024-05-26T20:18:56.999199+0000","last_became_peered":"2024-05-26T20:18:56.999199+0000","last_unstale":"2024-05-26T20:18:57.001075+0000","last_undegraded":"2024-05-26T20:18:57.001075+0000","last_fullsized":"2024-05-26T20:18:57.001075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:21:09.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.997094+0000","last_change":"2024-05-26T20:18:57.002096+0000","last_active":"2024-05-26T20:18:57.997094+0000","last_peered":"2024-05-26T20:18:57.997094+0000","last_clean":"2024-05-26T20:18:57.997094+0000","last_became_active":"2024-05-26T20:18:57.001816+0000","last_became_peered":"2024-05-26T20:18:57.001816+0000","last_unstale":"2024-05-26T20:18:57.997094+0000","last_undegraded":"2024-05-26T20:18:57.997094+0000","last_fullsized":"2024-05-26T20:18:57.997094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:29.151097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.003641+0000","last_change":"2024-05-26T20:18:57.002128+0000","last_active":"2024-05-26T20:18:57.003641+0000","last_peered":"2024-05-26T20:18:57.003641+0000","last_clean":"2024-05-26T20:18:57.003641+0000","last_became_active":"2024-05-26T20:18:57.001875+0000","last_became_peered":"2024-05-26T20:18:57.001875+0000","last_unstale":"2024-05-26T20:18:57.003641+0000","last_undegraded":"2024-05-26T20:18:57.003641+0000","last_fullsized":"2024-05-26T20:18:57.003641+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:41:21.946315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.000998+0000","last_change":"2024-05-26T20:18:56.999419+0000","last_active":"2024-05-26T20:18:57.000998+0000","last_peered":"2024-05-26T20:18:57.000998+0000","last_clean":"2024-05-26T20:18:57.000998+0000","last_became_active":"2024-05-26T20:18:56.999125+0000","last_became_peered":"2024-05-26T20:18:56.999125+0000","last_unstale":"2024-05-26T20:18:57.000998+0000","last_undegraded":"2024-05-26T20:18:57.000998+0000","last_fullsized":"2024-05-26T20:18:57.000998+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:54:51.122384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":98304,"data_stored":24981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T20:19:00.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-05-26T20:19:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-05-26T20:19:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-05-26T20:18:58.987708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82880,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489920,"statfs":{"total":322122547200,"available":322037678080,"internally_reserved":0,"allocated":1482752,"data_stored":1255679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.003062+0000","last_change":"2024-05-26T20:18:57.002225+0000","last_active":"2024-05-26T20:18:57.003062+0000","last_peered":"2024-05-26T20:18:57.003062+0000","last_clean":"2024-05-26T20:18:57.003062+0000","last_became_active":"2024-05-26T20:18:57.001923+0000","last_became_peered":"2024-05-26T20:18:57.001923+0000","last_unstale":"2024-05-26T20:18:57.003062+0000","last_undegraded":"2024-05-26T20:18:57.003062+0000","last_fullsized":"2024-05-26T20:18:57.003062+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:05:43.587224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.001014+0000","last_change":"2024-05-26T20:18:56.999962+0000","last_active":"2024-05-26T20:18:57.001014+0000","last_peered":"2024-05-26T20:18:57.001014+0000","last_clean":"2024-05-26T20:18:57.001014+0000","last_became_active":"2024-05-26T20:18:56.999651+0000","last_became_peered":"2024-05-26T20:18:56.999651+0000","last_unstale":"2024-05-26T20:18:57.001014+0000","last_undegraded":"2024-05-26T20:18:57.001014+0000","last_fullsized":"2024-05-26T20:18:57.001014+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:38.570740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.002033+0000","last_change":"2024-05-26T20:18:57.001156+0000","last_active":"2024-05-26T20:18:57.002033+0000","last_peered":"2024-05-26T20:18:57.002033+0000","last_clean":"2024-05-26T20:18:57.002033+0000","last_became_active":"2024-05-26T20:18:57.000872+0000","last_became_peered":"2024-05-26T20:18:57.000872+0000","last_unstale":"2024-05-26T20:18:57.002033+0000","last_undegraded":"2024-05-26T20:18:57.002033+0000","last_fullsized":"2024-05-26T20:18:57.002033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:25:26.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.997080+0000","last_change":"2024-05-26T20:18:57.002192+0000","last_active":"2024-05-26T20:18:57.997080+0000","last_peered":"2024-05-26T20:18:57.997080+0000","last_clean":"2024-05-26T20:18:57.997080+0000","last_became_active":"2024-05-26T20:18:57.001799+0000","last_became_peered":"2024-05-26T20:18:57.001799+0000","last_unstale":"2024-05-26T20:18:57.997080+0000","last_undegraded":"2024-05-26T20:18:57.997080+0000","last_fullsized":"2024-05-26T20:18:57.997080+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:41:21.946315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.492994+0000","last_change":"2024-05-26T20:18:57.003988+0000","last_active":"2024-05-26T20:18:57.492994+0000","last_peered":"2024-05-26T20:18:57.492994+0000","last_clean":"2024-05-26T20:18:57.492994+0000","last_became_active":"2024-05-26T20:18:57.003697+0000","last_became_peered":"2024-05-26T20:18:57.003697+0000","last_unstale":"2024-05-26T20:18:57.492994+0000","last_undegraded":"2024-05-26T20:18:57.492994+0000","last_fullsized":"2024-05-26T20:18:57.492994+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:46:27.924057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.001075+0000","last_change":"2024-05-26T20:18:56.999481+0000","last_active":"2024-05-26T20:18:57.001075+0000","last_peered":"2024-05-26T20:18:57.001075+0000","last_clean":"2024-05-26T20:18:57.001075+0000","last_became_active":"2024-05-26T20:18:56.999199+0000","last_became_peered":"2024-05-26T20:18:56.999199+0000","last_unstale":"2024-05-26T20:18:57.001075+0000","last_undegraded":"2024-05-26T20:18:57.001075+0000","last_fullsized":"2024-05-26T20:18:57.001075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:21:09.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.997094+0000","last_change":"2024-05-26T20:18:57.002096+0000","last_active":"2024-05-26T20:18:57.997094+0000","last_peered":"2024-05-26T20:18:57.997094+0000","last_clean":"2024-05-26T20:18:57.997094+0000","last_became_active":"2024-05-26T20:18:57.001816+0000","last_became_peered":"2024-05-26T20:18:57.001816+0000","last_unstale":"2024-05-26T20:18:57.997094+0000","last_undegraded":"2024-05-26T20:18:57.997094+0000","last_fullsized":"2024-05-26T20:18:57.997094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:29.151097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.003641+0000","last_change":"2024-05-26T20:18:57.002128+0000","last_active":"2024-05-26T20:18:57.003641+0000","last_peered":"2024-05-26T20:18:57.003641+0000","last_clean":"2024-05-26T20:18:57.003641+0000","last_became_active":"2024-05-26T20:18:57.001875+0000","last_became_peered":"2024-05-26T20:18:57.001875+0000","last_unstale":"2024-05-26T20:18:57.003641+0000","last_undegraded":"2024-05-26T20:18:57.003641+0000","last_fullsized":"2024-05-26T20:18:57.003641+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:41:21.946315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:18:57.000998+0000","last_change":"2024-05-26T20:18:56.999419+0000","last_active":"2024-05-26T20:18:57.000998+0000","last_peered":"2024-05-26T20:18:57.000998+0000","last_clean":"2024-05-26T20:18:57.000998+0000","last_became_active":"2024-05-26T20:18:56.999125+0000","last_became_peered":"2024-05-26T20:18:56.999125+0000","last_unstale":"2024-05-26T20:18:57.000998+0000","last_undegraded":"2024-05-26T20:18:57.000998+0000","last_fullsized":"2024-05-26T20:18:57.000998+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:18:55.986700+0000","last_clean_scrub_stamp":"2024-05-26T20:18:55.986700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:54:51.122384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":98304,"data_stored":24981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T20:19:00.963 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-05-26T20:19:00.963 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-05-26T20:19:00.963 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-05-26T20:19:00.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-05-26T20:19:01.349 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:01.349 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-26T20:19:01.366 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-05-26T20:19:01.366 INFO:teuthology.run_tasks:Running task ceph... 2024-05-26T20:19:01.376 INFO:tasks.ceph:Creating ceph cluster c2... 2024-05-26T20:19:01.376 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-26T20:19:01.376 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7725509', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1296, 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBCfhtYy3Qw3mwMAX9p72Gz0S7a+LiV/khZYrpw0xSead6z7IlHJ4uA9wVBZvncokQwdr9xZoMKI5ZkZ3+i33c=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFdhq4LVhtbDhGEnFztzQHEDXgbHB7CDfQGa/mAGD0p7mZ1B2lRDq+A7AtI9SE0/anQeadF/vf9q0uiaPCFTsQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'f1882d8093fb4b60c1d1e52444565787ca4bd6cc', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e7ae7580c417d6ab27fda98fecadca5d9f85d63e', 'timestamp': '2024-05-24_22:40:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-26T20:19:01.376 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-05-26T20:19:01.379 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-05-26T20:19:01.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-05-26T20:19:01.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-05-26T20:19:01.474 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:01.474 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-05-26T20:19:01.534 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-26T20:19:01.534 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-05-26T20:19:01.594 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 168887 Links: 1 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-05-26 20:18:27.640035203 +0000 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-05-26 20:13:57.074086537 +0000 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-05-26 20:13:57.074086537 +0000 2024-05-26T20:19:01.595 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-05-26T20:19:01.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-26T20:19:01.662 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-05-26T20:19:01.662 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-05-26T20:19:01.663 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000258383 s, 2.0 MB/s 2024-05-26T20:19:01.664 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-26T20:19:01.724 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-05-26T20:19:01.782 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 174312 Links: 1 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-05-26 20:18:27.640035203 +0000 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-05-26 20:13:57.411080724 +0000 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-05-26 20:13:57.411080724 +0000 2024-05-26T20:19:01.783 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-05-26T20:19:01.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-26T20:19:01.851 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-05-26T20:19:01.851 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-05-26T20:19:01.851 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000239216 s, 2.1 MB/s 2024-05-26T20:19:01.851 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-26T20:19:01.910 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 173301 Links: 1 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:19:01.968 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-05-26 20:18:27.640035203 +0000 2024-05-26T20:19:01.969 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-05-26 20:13:57.770074532 +0000 2024-05-26T20:19:01.969 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-05-26 20:13:57.770074532 +0000 2024-05-26T20:19:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-05-26T20:19:01.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-26T20:19:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-05-26T20:19:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-05-26T20:19:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.0001911 s, 2.7 MB/s 2024-05-26T20:19:02.036 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-26T20:19:02.095 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 178243 Links: 1 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-05-26 20:18:27.640035203 +0000 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-05-26 20:13:58.146068047 +0000 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-05-26 20:13:58.146068047 +0000 2024-05-26T20:19:02.153 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-05-26T20:19:02.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-26T20:19:02.219 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-05-26T20:19:02.219 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-05-26T20:19:02.219 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000207727 s, 2.5 MB/s 2024-05-26T20:19:02.220 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-26T20:19:02.278 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:19:02.278 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-26T20:19:02.279 INFO:tasks.ceph:Generating config... 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] debug rgw = 20 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-05-26T20:19:02.280 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-05-26T20:19:02.281 INFO:tasks.ceph:[client] setgroup = ceph 2024-05-26T20:19:02.281 INFO:tasks.ceph:[client] setuser = ceph 2024-05-26T20:19:02.281 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-05-26T20:19:02.281 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-26T20:19:02.281 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-26T20:19:02.281 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-26T20:19:02.281 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-26T20:19:02.281 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-05-26T20:19:02.281 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-05-26T20:19:02.282 INFO:tasks.ceph:Setting up c2.mon.a... 2024-05-26T20:19:02.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-05-26T20:19:02.376 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.keyring 2024-05-26T20:19:02.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-05-26T20:19:02.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-05-26T20:19:02.449 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.137')] 2024-05-26T20:19:02.449 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.137'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-05-26T20:19:02.450 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:02.450 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-26T20:19:02.507 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.137 --print /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:02.588 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:02.588 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid 5cda2af0-8c62-4a98-a447-09e503ef5056 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = pacific 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:fsid 5cda2af0-8c62-4a98-a447-09e503ef5056 2024-05-26T20:19:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2024-05-26T20:19:02.590579+0000 2024-05-26T20:19:02.590 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-05-26T20:19:02.590579+0000 2024-05-26T20:19:02.590 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 16 (pacific) 2024-05-26T20:19:02.590 INFO:teuthology.orchestra.run.smithi137.stdout:election_strategy: 1 2024-05-26T20:19:02.590 INFO:teuthology.orchestra.run.smithi137.stdout:0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.a 2024-05-26T20:19:02.590 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-05-26T20:19:02.594 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-26T20:19:02.651 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 5cda2af0-8c62-4a98-a447-09e503ef5056... 2024-05-26T20:19:02.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-05-26T20:19:02.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-05-26T20:19:02.706 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.707 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-05-26T20:19:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.137 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 5cda2af0-8c62-4a98-a447-09e503ef5056 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.710 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-05-26T20:19:02.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: rocksdb delete range threshold = 0 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-05-26T20:19:02.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-05-26T20:19:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-05-26T20:19:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-26T20:19:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw = 20 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw sync = 20 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt require ssl = False 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt s3 kms backend = testing 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw curl low speed time = 300 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw data log num shards = 4 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw md log max shards = 4 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync data inject err probability = 0.1 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync log trim interval = 0 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync meta inject err probability = 0.1 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: rgw sync obj etag verify = True 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: setgroup = ceph 2024-05-26T20:19:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: setuser = ceph 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: rgw data notify interval msec = 0 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 10 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 1000 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-05-26T20:19:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: osd_max_omap_entries_per_request = 10000 2024-05-26T20:19:02.734 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-05-26T20:19:02.735 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-05-26T20:19:02.736 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-05-26T20:19:02.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.137 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 5cda2af0-8c62-4a98-a447-09e503ef5056 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-05-26T20:19:02.738 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.739 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:19:02.740 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-05-26T20:19:02.741 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.742 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw sync = 20 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2024-05-26T20:19:02.743 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync data inject err probability = 0.1 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync meta inject err probability = 0.1 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync obj etag verify = True 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data notify interval msec = 0 2024-05-26T20:19:02.744 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-05-26T20:19:02.745 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2024-05-26T20:19:02.745 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-05-26T20:19:02.745 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2024-05-26T20:19:02.745 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-05-26T20:19:02.745 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2024-05-26T20:19:02.746 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-05-26T20:19:02.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-05-26T20:19:02.842 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-26T20:19:02.843 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:02.843 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-05-26T20:19:02.860 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:02.860 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-05-26T20:19:02.917 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T20:19:02.917 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:19:02.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/c2.keyring 2024-05-26T20:19:02.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-05-26T20:19:02.962 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-05-26T20:19:02.962 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:03.023 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:19:03.023 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:03.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/c2.keyring 2024-05-26T20:19:03.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-05-26T20:19:03.059 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:03.059 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:03.116 INFO:tasks.ceph:Setting up mon nodes... 2024-05-26T20:19:03.116 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-26T20:19:03.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-05-26T20:19:03.209 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-05-26T20:19:03.211 INFO:tasks.ceph:Setting up mds nodes... 2024-05-26T20:19:03.211 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-26T20:19:03.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-05-26T20:19:03.254 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.client.0.keyring 2024-05-26T20:19:03.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-05-26T20:19:03.350 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.client.1.keyring 2024-05-26T20:19:03.361 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-26T20:19:03.362 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-26T20:19:03.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-05-26T20:19:03.429 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:19:03.429 INFO:tasks.ceph:role: c2.osd.0 2024-05-26T20:19:03.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:19:03.429 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:19:03.503 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:19:03.932 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-05-26T20:19:03.934 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-05-26T20:19:03.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-05-26T20:19:04.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-05-26T20:19:04.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-05-26T20:19:04.188 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:19:04.188 INFO:tasks.ceph:role: c2.osd.1 2024-05-26T20:19:04.188 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:19:04.188 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:19:04.262 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:19:04.263 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:19:04.689 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-05-26T20:19:04.690 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-05-26T20:19:04.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-05-26T20:19:04.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-05-26T20:19:04.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-05-26T20:19:04.891 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-26T20:19:04.892 INFO:tasks.ceph:role: c2.osd.2 2024-05-26T20:19:04.892 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T20:19:04.892 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-26T20:19:04.965 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-26T20:19:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-26T20:19:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-26T20:19:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-26T20:19:04.966 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-26T20:19:05.415 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-05-26T20:19:05.416 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-05-26T20:19:05.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-05-26T20:19:05.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-05-26T20:19:05.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:05.712 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.712+0000 7f6f05eec7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-05-26T20:19:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.716+0000 7f6f05eec7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-05-26T20:19:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.716+0000 7f6f05eec7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-05-26T20:19:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.716+0000 7f6f05eec7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-05-26T20:19:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.716+0000 7f6f05eec7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-05-26T20:19:05.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:05.723+0000 7f6f05eec7c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-05-26T20:19:07.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-05-26T20:19:07.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.918+0000 7ff21b0807c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.918+0000 7ff21b0807c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.918+0000 7ff21b0807c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.918+0000 7ff21b0807c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.918+0000 7ff21b0807c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-05-26T20:19:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:07.919+0000 7ff21b0807c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-05-26T20:19:10.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-05-26T20:19:10.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:10.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.125+0000 7f4ff05457c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-05-26T20:19:10.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.126+0000 7f4ff05457c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-05-26T20:19:10.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.126+0000 7f4ff05457c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-05-26T20:19:10.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.126+0000 7f4ff05457c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-05-26T20:19:10.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.126+0000 7f4ff05457c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-05-26T20:19:10.126 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:10.126+0000 7f4ff05457c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-05-26T20:19:12.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-05-26T20:19:12.233 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-26T20:19:12.233 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-05-26T20:19:12.302 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-05-26T20:19:12.370 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-05-26T20:19:12.438 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-05-26T20:19:12.505 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.505 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-05-26T20:19:12.563 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:12.563 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-05-26T20:19:12.620 INFO:tasks.ceph:Adding keys to all mons... 2024-05-26T20:19:12.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/c2.keyring 2024-05-26T20:19:12.685 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-05-26T20:19:12.685 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA3mVNmuHaIDBAA+AABJvLjTPH5ALx7JO3tAw== 2024-05-26T20:19:12.685 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA5mVNmSxySKhAAA5xbe4jfNh3JlBYbnC01Dw== 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA7mVNmue3SNhAAXj08C9yD1jP+tu+lV1Yl8A== 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA+mVNm0cOGBxAAJrEF3c9ExCibZnsG56gaOg== 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA3mVNme0xCDxAAUYiY66/cC3VwQ8Kpp69kdA== 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-05-26T20:19:12.686 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA3mVNm4MT0FBAAr6jet9BEJHO/yLLUlyFiQA== 2024-05-26T20:19:12.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-26T20:19:12.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:19:12.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:19:12.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-26T20:19:12.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-26T20:19:12.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-26T20:19:13.051 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-26T20:19:13.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-05-26T20:19:13.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-05-26T20:19:13.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-05-26T20:19:13.204 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-05-26T20:19:13.263 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-05-26T20:19:13.264 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-05-26T20:19:13.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-05-26T20:19:13.306 INFO:tasks.ceph.c2.mon.a:Started 2024-05-26T20:19:13.306 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-05-26T20:19:13.307 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-05-26T20:19:13.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-05-26T20:19:13.309 INFO:tasks.ceph.c2.mgr.x:Started 2024-05-26T20:19:13.309 INFO:tasks.ceph:Setting crush tunables to default 2024-05-26T20:19:13.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd crush tunables default 2024-05-26T20:19:13.515 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-05-26T20:19:13.526 INFO:tasks.ceph:check_enable_crimson: False 2024-05-26T20:19:13.526 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-05-26T20:19:13.526 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:13.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-05-26T20:19:13.553 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:13.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-05-26T20:19:13.620 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-05-26T20:19:13.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-05-26T20:19:13.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new a94c6d85-201c-40b7-b1dc-07d69d6d32b0 0 2024-05-26T20:19:13.709 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:13.710+0000 7f68e0214200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T20:19:13.886 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:13.887+0000 7f68e0214200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T20:19:13.964 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-05-26T20:19:13.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new e747afc8-0e5c-4caa-9d48-fb3cc272c47a 1 2024-05-26T20:19:14.188 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-05-26T20:19:14.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d 2 2024-05-26T20:19:14.256 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:14.257+0000 7f68e0214200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T20:19:14.415 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-05-26T20:19:14.427 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-05-26T20:19:14.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-05-26T20:19:14.430 INFO:tasks.ceph.c2.osd.0:Started 2024-05-26T20:19:14.431 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-05-26T20:19:14.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-05-26T20:19:14.433 INFO:tasks.ceph.c2.osd.1:Started 2024-05-26T20:19:14.433 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-05-26T20:19:14.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-05-26T20:19:14.435 INFO:tasks.ceph.c2.osd.2:Started 2024-05-26T20:19:14.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-05-26T20:19:14.644 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:14.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":5,"fsid":"5cda2af0-8c62-4a98-a447-09e503ef5056","created":"2024-05-26T20:19:13.419690+0000","modified":"2024-05-26T20:19:14.414648+0000","last_up_change":"0.000000","last_in_change":"2024-05-26T20:19:14.414648+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a94c6d85-201c-40b7-b1dc-07d69d6d32b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e747afc8-0e5c-4caa-9d48-fb3cc272c47a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:19:14.656 INFO:tasks.ceph.ceph_manager.c2:[] 2024-05-26T20:19:14.656 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-26T20:19:14.857 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:14.858+0000 7f68e0214200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T20:19:14.957 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-05-26T20:19:15.032 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-05-26T20:19:15.033+0000 7f3f6747a7c0 -1 Falling back to public interface 2024-05-26T20:19:15.039 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-05-26T20:19:15.040+0000 7f5aac7717c0 -1 Falling back to public interface 2024-05-26T20:19:15.067 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-05-26T20:19:15.068+0000 7faf984467c0 -1 Falling back to public interface 2024-05-26T20:19:15.071 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:15.072+0000 7f68e0214200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T20:19:15.151 INFO:teuthology.misc.health.smithi137.stdout: 2024-05-26T20:19:15.151 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":5,"fsid":"5cda2af0-8c62-4a98-a447-09e503ef5056","created":"2024-05-26T20:19:13.419690+0000","modified":"2024-05-26T20:19:14.414648+0000","last_up_change":"0.000000","last_in_change":"2024-05-26T20:19:14.414648+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a94c6d85-201c-40b7-b1dc-07d69d6d32b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e747afc8-0e5c-4caa-9d48-fb3cc272c47a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:19:15.162 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-26T20:19:15.462 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:15.463+0000 7f68e0214200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T20:19:16.138 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.139+0000 7f68e0214200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T20:19:16.212 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.213+0000 7f68e0214200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T20:19:16.344 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.345+0000 7f68e0214200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T20:19:16.521 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.522+0000 7f68e0214200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T20:19:16.593 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.594+0000 7f68e0214200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T20:19:16.671 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.672+0000 7f68e0214200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T20:19:16.747 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.748+0000 7f68e0214200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T20:19:16.828 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:16.829+0000 7f68e0214200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T20:19:17.040 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:17.041+0000 7f68e0214200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T20:19:17.256 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:17.257+0000 7f68e0214200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T20:19:17.411 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:17.412+0000 7f68e0214200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T20:19:17.484 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:17.485+0000 7f68e0214200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T20:19:17.726 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-05-26T20:19:17.727+0000 7f3f6747a7c0 -1 osd.0 0 log_to_monitors true 2024-05-26T20:19:17.762 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-05-26T20:19:17.763+0000 7f5aac7717c0 -1 osd.2 0 log_to_monitors true 2024-05-26T20:19:17.795 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-05-26T20:19:17.796+0000 7faf984467c0 -1 osd.1 0 log_to_monitors true 2024-05-26T20:19:18.167 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:18.168+0000 7f68e0214200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T20:19:18.330 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:18.331+0000 7f68e0214200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T20:19:18.489 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:18.490+0000 7f68e0214200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T20:19:18.906 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:18.906+0000 7f68e0214200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T20:19:19.086 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:19.087+0000 7f68e0214200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T20:19:19.499 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:19.500+0000 7f68e0214200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T20:19:19.629 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:19.629+0000 7f68e0214200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T20:19:19.649 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-05-26T20:19:19.650+0000 7f3f5b81c700 -1 osd.0 0 waiting for initial osdmap 2024-05-26T20:19:19.654 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-05-26T20:19:19.655+0000 7f5aa0b13700 -1 osd.2 0 waiting for initial osdmap 2024-05-26T20:19:19.657 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-05-26T20:19:19.658+0000 7f3f533f6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:19:19.675 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-05-26T20:19:19.676+0000 7f5a986ed700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:19:20.266 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-05-26T20:19:20.266+0000 7faf8c7e8700 -1 osd.1 0 waiting for initial osdmap 2024-05-26T20:19:20.279 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-05-26T20:19:20.280+0000 7faf843c2700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T20:19:20.661 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-05-26T20:19:20.662+0000 7f68a15fe700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-26T20:19:21.465 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-05-26T20:19:21.794 INFO:teuthology.misc.health.smithi137.stdout: 2024-05-26T20:19:21.794 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":9,"fsid":"5cda2af0-8c62-4a98-a447-09e503ef5056","created":"2024-05-26T20:19:13.419690+0000","modified":"2024-05-26T20:19:21.267976+0000","last_up_change":"2024-05-26T20:19:21.267976+0000","last_in_change":"2024-05-26T20:19:14.414648+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a94c6d85-201c-40b7-b1dc-07d69d6d32b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":42104},{"type":"v1","addr":"172.21.15.137:6801","nonce":42104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":42104},{"type":"v1","addr":"172.21.15.137:6803","nonce":42104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":42104},{"type":"v1","addr":"172.21.15.137:6807","nonce":42104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":42104},{"type":"v1","addr":"172.21.15.137:6805","nonce":42104}]},"public_addr":"172.21.15.137:6801/42104","cluster_addr":"172.21.15.137:6803/42104","heartbeat_back_addr":"172.21.15.137:6807/42104","heartbeat_front_addr":"172.21.15.137:6805/42104","state":["exists","up"]},{"osd":1,"uuid":"e747afc8-0e5c-4caa-9d48-fb3cc272c47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":42126},{"type":"v1","addr":"172.21.15.137:6817","nonce":42126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":42126},{"type":"v1","addr":"172.21.15.137:6819","nonce":42126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":42126},{"type":"v1","addr":"172.21.15.137:6823","nonce":42126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":42126},{"type":"v1","addr":"172.21.15.137:6821","nonce":42126}]},"public_addr":"172.21.15.137:6817/42126","cluster_addr":"172.21.15.137:6819/42126","heartbeat_back_addr":"172.21.15.137:6823/42126","heartbeat_front_addr":"172.21.15.137:6821/42126","state":["exists","up"]},{"osd":2,"uuid":"b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":42105},{"type":"v1","addr":"172.21.15.137:6809","nonce":42105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":42105},{"type":"v1","addr":"172.21.15.137:6811","nonce":42105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":42105},{"type":"v1","addr":"172.21.15.137:6815","nonce":42105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":42105},{"type":"v1","addr":"172.21.15.137:6813","nonce":42105}]},"public_addr":"172.21.15.137:6809/42105","cluster_addr":"172.21.15.137:6811/42105","heartbeat_back_addr":"172.21.15.137:6815/42105","heartbeat_front_addr":"172.21.15.137:6813/42105","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.698518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.744794+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:19:21.807 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-26T20:19:21.807 INFO:tasks.ceph:Creating RBD pool 2024-05-26T20:19:21.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd pool create rbd 8 2024-05-26T20:19:22.276 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'rbd' created 2024-05-26T20:19:22.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-26T20:19:23.276 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-26T20:19:23.291 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-05-26T20:19:23.291 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-05-26T20:19:23.292 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-05-26T20:19:23.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-05-26T20:19:23.292 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-26T20:19:23.714 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:23.727 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":41692},{"type":"v1","addr":"172.21.15.137:6825","nonce":41692}]},"active_addr":"172.21.15.137:6825/41692","active_change":"2024-05-26T20:19:19.643275+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3125623397}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2049085464}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1085781521}]}]} 2024-05-26T20:19:23.730 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-05-26T20:19:23.730 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-05-26T20:19:23.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-05-26T20:19:24.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:24.052 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":11,"fsid":"5cda2af0-8c62-4a98-a447-09e503ef5056","created":"2024-05-26T20:19:13.419690+0000","modified":"2024-05-26T20:19:23.271638+0000","last_up_change":"2024-05-26T20:19:21.267976+0000","last_in_change":"2024-05-26T20:19:14.414648+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T20:19:21.689065+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-26T20:19:22.143991+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a94c6d85-201c-40b7-b1dc-07d69d6d32b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":42104},{"type":"v1","addr":"172.21.15.137:6801","nonce":42104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":42104},{"type":"v1","addr":"172.21.15.137:6803","nonce":42104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":42104},{"type":"v1","addr":"172.21.15.137:6807","nonce":42104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":42104},{"type":"v1","addr":"172.21.15.137:6805","nonce":42104}]},"public_addr":"172.21.15.137:6801/42104","cluster_addr":"172.21.15.137:6803/42104","heartbeat_back_addr":"172.21.15.137:6807/42104","heartbeat_front_addr":"172.21.15.137:6805/42104","state":["exists","up"]},{"osd":1,"uuid":"e747afc8-0e5c-4caa-9d48-fb3cc272c47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":42126},{"type":"v1","addr":"172.21.15.137:6817","nonce":42126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":42126},{"type":"v1","addr":"172.21.15.137:6819","nonce":42126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":42126},{"type":"v1","addr":"172.21.15.137:6823","nonce":42126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":42126},{"type":"v1","addr":"172.21.15.137:6821","nonce":42126}]},"public_addr":"172.21.15.137:6817/42126","cluster_addr":"172.21.15.137:6819/42126","heartbeat_back_addr":"172.21.15.137:6823/42126","heartbeat_front_addr":"172.21.15.137:6821/42126","state":["exists","up"]},{"osd":2,"uuid":"b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":42105},{"type":"v1","addr":"172.21.15.137:6809","nonce":42105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":42105},{"type":"v1","addr":"172.21.15.137:6811","nonce":42105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":42105},{"type":"v1","addr":"172.21.15.137:6815","nonce":42105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":42105},{"type":"v1","addr":"172.21.15.137:6813","nonce":42105}]},"public_addr":"172.21.15.137:6809/42105","cluster_addr":"172.21.15.137:6811/42105","heartbeat_back_addr":"172.21.15.137:6815/42105","heartbeat_front_addr":"172.21.15.137:6813/42105","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.698518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.813700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.744794+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:19:24.066 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-05-26T20:19:24.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-05-26T20:19:24.391 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:24.391 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":12,"fsid":"5cda2af0-8c62-4a98-a447-09e503ef5056","created":"2024-05-26T20:19:13.419690+0000","modified":"2024-05-26T20:19:24.272549+0000","last_up_change":"2024-05-26T20:19:21.267976+0000","last_in_change":"2024-05-26T20:19:14.414648+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T20:19:21.689065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-26T20:19:22.143991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a94c6d85-201c-40b7-b1dc-07d69d6d32b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":42104},{"type":"v1","addr":"172.21.15.137:6801","nonce":42104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":42104},{"type":"v1","addr":"172.21.15.137:6803","nonce":42104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":42104},{"type":"v1","addr":"172.21.15.137:6807","nonce":42104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":42104},{"type":"v1","addr":"172.21.15.137:6805","nonce":42104}]},"public_addr":"172.21.15.137:6801/42104","cluster_addr":"172.21.15.137:6803/42104","heartbeat_back_addr":"172.21.15.137:6807/42104","heartbeat_front_addr":"172.21.15.137:6805/42104","state":["exists","up"]},{"osd":1,"uuid":"e747afc8-0e5c-4caa-9d48-fb3cc272c47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":42126},{"type":"v1","addr":"172.21.15.137:6817","nonce":42126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":42126},{"type":"v1","addr":"172.21.15.137:6819","nonce":42126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":42126},{"type":"v1","addr":"172.21.15.137:6823","nonce":42126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":42126},{"type":"v1","addr":"172.21.15.137:6821","nonce":42126}]},"public_addr":"172.21.15.137:6817/42126","cluster_addr":"172.21.15.137:6819/42126","heartbeat_back_addr":"172.21.15.137:6823/42126","heartbeat_front_addr":"172.21.15.137:6821/42126","state":["exists","up"]},{"osd":2,"uuid":"b2dd6911-d6cb-4237-9cb2-0dc3f2fe773d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":42105},{"type":"v1","addr":"172.21.15.137:6809","nonce":42105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":42105},{"type":"v1","addr":"172.21.15.137:6811","nonce":42105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":42105},{"type":"v1","addr":"172.21.15.137:6815","nonce":42105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":42105},{"type":"v1","addr":"172.21.15.137:6813","nonce":42105}]},"public_addr":"172.21.15.137:6809/42105","cluster_addr":"172.21.15.137:6811/42105","heartbeat_back_addr":"172.21.15.137:6815/42105","heartbeat_front_addr":"172.21.15.137:6813/42105","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.698518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.813700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T20:19:18.744794+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T20:19:24.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-05-26T20:19:24.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-05-26T20:19:24.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-05-26T20:19:24.574 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-05-26T20:19:24.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-05-26T20:19:24.577 INFO:teuthology.orchestra.run.smithi137.stdout:38654705667 2024-05-26T20:19:24.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-05-26T20:19:24.587 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-05-26T20:19:24.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-05-26T20:19:24.918 INFO:teuthology.orchestra.run.smithi137.stdout:34359738369 2024-05-26T20:19:24.932 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-05-26T20:19:24.972 INFO:teuthology.orchestra.run.smithi137.stdout:38654705666 2024-05-26T20:19:24.980 INFO:teuthology.orchestra.run.smithi137.stdout:34359738369 2024-05-26T20:19:24.986 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.1 2024-05-26T20:19:24.995 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2024-05-26T20:19:25.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-05-26T20:19:25.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-05-26T20:19:25.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-05-26T20:19:26.288 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-05-26T20:19:26.303 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2024-05-26T20:19:26.303 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:26.334 INFO:teuthology.orchestra.run.smithi137.stdout:38654705667 2024-05-26T20:19:26.348 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.1 2024-05-26T20:19:26.348 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:26.376 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-05-26T20:19:26.390 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2024-05-26T20:19:26.390 DEBUG:teuthology.parallel:result is None 2024-05-26T20:19:26.390 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-05-26T20:19:26.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-05-26T20:19:26.750 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:26.752 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-05-26T20:19:26.763 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-05-26T20:19:25.648595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82688,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490112,"statfs":{"total":322122547200,"available":322037874688,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289141+0000","last_change":"2024-05-26T20:19:23.282238+0000","last_active":"2024-05-26T20:19:23.289141+0000","last_peered":"2024-05-26T20:19:23.289141+0000","last_clean":"2024-05-26T20:19:23.289141+0000","last_became_active":"2024-05-26T20:19:23.281988+0000","last_became_peered":"2024-05-26T20:19:23.281988+0000","last_unstale":"2024-05-26T20:19:23.289141+0000","last_undegraded":"2024-05-26T20:19:23.289141+0000","last_fullsized":"2024-05-26T20:19:23.289141+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:57:28.553550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289318+0000","last_change":"2024-05-26T20:19:23.284547+0000","last_active":"2024-05-26T20:19:23.289318+0000","last_peered":"2024-05-26T20:19:23.289318+0000","last_clean":"2024-05-26T20:19:23.289318+0000","last_became_active":"2024-05-26T20:19:23.284315+0000","last_became_peered":"2024-05-26T20:19:23.284315+0000","last_unstale":"2024-05-26T20:19:23.289318+0000","last_undegraded":"2024-05-26T20:19:23.289318+0000","last_fullsized":"2024-05-26T20:19:23.289318+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:30:13.964652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289299+0000","last_change":"2024-05-26T20:19:23.284536+0000","last_active":"2024-05-26T20:19:23.289299+0000","last_peered":"2024-05-26T20:19:23.289299+0000","last_clean":"2024-05-26T20:19:23.289299+0000","last_became_active":"2024-05-26T20:19:23.284291+0000","last_became_peered":"2024-05-26T20:19:23.284291+0000","last_unstale":"2024-05-26T20:19:23.289299+0000","last_undegraded":"2024-05-26T20:19:23.289299+0000","last_fullsized":"2024-05-26T20:19:23.289299+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:44:53.399472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:19:24.277472+0000","last_change":"2024-05-26T20:19:23.285040+0000","last_active":"2024-05-26T20:19:24.277472+0000","last_peered":"2024-05-26T20:19:24.277472+0000","last_clean":"2024-05-26T20:19:24.277472+0000","last_became_active":"2024-05-26T20:19:23.284787+0000","last_became_peered":"2024-05-26T20:19:23.284787+0000","last_unstale":"2024-05-26T20:19:24.277472+0000","last_undegraded":"2024-05-26T20:19:24.277472+0000","last_fullsized":"2024-05-26T20:19:24.277472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:29:26.396929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.683439+0000","last_change":"2024-05-26T20:19:23.281981+0000","last_active":"2024-05-26T20:19:23.683439+0000","last_peered":"2024-05-26T20:19:23.683439+0000","last_clean":"2024-05-26T20:19:23.683439+0000","last_became_active":"2024-05-26T20:19:23.281723+0000","last_became_peered":"2024-05-26T20:19:23.281723+0000","last_unstale":"2024-05-26T20:19:23.683439+0000","last_undegraded":"2024-05-26T20:19:23.683439+0000","last_fullsized":"2024-05-26T20:19:23.683439+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:48:20.120754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.288986+0000","last_change":"2024-05-26T20:19:23.281833+0000","last_active":"2024-05-26T20:19:23.288986+0000","last_peered":"2024-05-26T20:19:23.288986+0000","last_clean":"2024-05-26T20:19:23.288986+0000","last_became_active":"2024-05-26T20:19:23.281582+0000","last_became_peered":"2024-05-26T20:19:23.281582+0000","last_unstale":"2024-05-26T20:19:23.288986+0000","last_undegraded":"2024-05-26T20:19:23.288986+0000","last_fullsized":"2024-05-26T20:19:23.288986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:56:43.670189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:19:24.277460+0000","last_change":"2024-05-26T20:19:23.284447+0000","last_active":"2024-05-26T20:19:24.277460+0000","last_peered":"2024-05-26T20:19:24.277460+0000","last_clean":"2024-05-26T20:19:24.277460+0000","last_became_active":"2024-05-26T20:19:23.284184+0000","last_became_peered":"2024-05-26T20:19:23.284184+0000","last_unstale":"2024-05-26T20:19:24.277460+0000","last_undegraded":"2024-05-26T20:19:24.277460+0000","last_fullsized":"2024-05-26T20:19:24.277460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:59:09.766003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289211+0000","last_change":"2024-05-26T20:19:23.284431+0000","last_active":"2024-05-26T20:19:23.289211+0000","last_peered":"2024-05-26T20:19:23.289211+0000","last_clean":"2024-05-26T20:19:23.289211+0000","last_became_active":"2024-05-26T20:19:23.284201+0000","last_became_peered":"2024-05-26T20:19:23.284201+0000","last_unstale":"2024-05-26T20:19:23.289211+0000","last_undegraded":"2024-05-26T20:19:23.289211+0000","last_fullsized":"2024-05-26T20:19:23.289211+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:56:14.324086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.288973+0000","last_change":"2024-05-26T20:19:23.281801+0000","last_active":"2024-05-26T20:19:23.288973+0000","last_peered":"2024-05-26T20:19:23.288973+0000","last_clean":"2024-05-26T20:19:23.288973+0000","last_became_active":"2024-05-26T20:19:23.281578+0000","last_became_peered":"2024-05-26T20:19:23.281578+0000","last_unstale":"2024-05-26T20:19:23.288973+0000","last_undegraded":"2024-05-26T20:19:23.288973+0000","last_fullsized":"2024-05-26T20:19:23.288973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:53:58.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T20:19:26.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-05-26T20:19:27.079 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:27.081 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-05-26T20:19:27.092 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-05-26T20:19:25.648595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82688,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490112,"statfs":{"total":322122547200,"available":322037874688,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289141+0000","last_change":"2024-05-26T20:19:23.282238+0000","last_active":"2024-05-26T20:19:23.289141+0000","last_peered":"2024-05-26T20:19:23.289141+0000","last_clean":"2024-05-26T20:19:23.289141+0000","last_became_active":"2024-05-26T20:19:23.281988+0000","last_became_peered":"2024-05-26T20:19:23.281988+0000","last_unstale":"2024-05-26T20:19:23.289141+0000","last_undegraded":"2024-05-26T20:19:23.289141+0000","last_fullsized":"2024-05-26T20:19:23.289141+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:57:28.553550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289318+0000","last_change":"2024-05-26T20:19:23.284547+0000","last_active":"2024-05-26T20:19:23.289318+0000","last_peered":"2024-05-26T20:19:23.289318+0000","last_clean":"2024-05-26T20:19:23.289318+0000","last_became_active":"2024-05-26T20:19:23.284315+0000","last_became_peered":"2024-05-26T20:19:23.284315+0000","last_unstale":"2024-05-26T20:19:23.289318+0000","last_undegraded":"2024-05-26T20:19:23.289318+0000","last_fullsized":"2024-05-26T20:19:23.289318+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:30:13.964652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289299+0000","last_change":"2024-05-26T20:19:23.284536+0000","last_active":"2024-05-26T20:19:23.289299+0000","last_peered":"2024-05-26T20:19:23.289299+0000","last_clean":"2024-05-26T20:19:23.289299+0000","last_became_active":"2024-05-26T20:19:23.284291+0000","last_became_peered":"2024-05-26T20:19:23.284291+0000","last_unstale":"2024-05-26T20:19:23.289299+0000","last_undegraded":"2024-05-26T20:19:23.289299+0000","last_fullsized":"2024-05-26T20:19:23.289299+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:44:53.399472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:19:24.277472+0000","last_change":"2024-05-26T20:19:23.285040+0000","last_active":"2024-05-26T20:19:24.277472+0000","last_peered":"2024-05-26T20:19:24.277472+0000","last_clean":"2024-05-26T20:19:24.277472+0000","last_became_active":"2024-05-26T20:19:23.284787+0000","last_became_peered":"2024-05-26T20:19:23.284787+0000","last_unstale":"2024-05-26T20:19:24.277472+0000","last_undegraded":"2024-05-26T20:19:24.277472+0000","last_fullsized":"2024-05-26T20:19:24.277472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:29:26.396929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.683439+0000","last_change":"2024-05-26T20:19:23.281981+0000","last_active":"2024-05-26T20:19:23.683439+0000","last_peered":"2024-05-26T20:19:23.683439+0000","last_clean":"2024-05-26T20:19:23.683439+0000","last_became_active":"2024-05-26T20:19:23.281723+0000","last_became_peered":"2024-05-26T20:19:23.281723+0000","last_unstale":"2024-05-26T20:19:23.683439+0000","last_undegraded":"2024-05-26T20:19:23.683439+0000","last_fullsized":"2024-05-26T20:19:23.683439+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:48:20.120754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.288986+0000","last_change":"2024-05-26T20:19:23.281833+0000","last_active":"2024-05-26T20:19:23.288986+0000","last_peered":"2024-05-26T20:19:23.288986+0000","last_clean":"2024-05-26T20:19:23.288986+0000","last_became_active":"2024-05-26T20:19:23.281582+0000","last_became_peered":"2024-05-26T20:19:23.281582+0000","last_unstale":"2024-05-26T20:19:23.288986+0000","last_undegraded":"2024-05-26T20:19:23.288986+0000","last_fullsized":"2024-05-26T20:19:23.288986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:56:43.670189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-26T20:19:24.277460+0000","last_change":"2024-05-26T20:19:23.284447+0000","last_active":"2024-05-26T20:19:24.277460+0000","last_peered":"2024-05-26T20:19:24.277460+0000","last_clean":"2024-05-26T20:19:24.277460+0000","last_became_active":"2024-05-26T20:19:23.284184+0000","last_became_peered":"2024-05-26T20:19:23.284184+0000","last_unstale":"2024-05-26T20:19:24.277460+0000","last_undegraded":"2024-05-26T20:19:24.277460+0000","last_fullsized":"2024-05-26T20:19:24.277460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:59:09.766003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.289211+0000","last_change":"2024-05-26T20:19:23.284431+0000","last_active":"2024-05-26T20:19:23.289211+0000","last_peered":"2024-05-26T20:19:23.289211+0000","last_clean":"2024-05-26T20:19:23.289211+0000","last_became_active":"2024-05-26T20:19:23.284201+0000","last_became_peered":"2024-05-26T20:19:23.284201+0000","last_unstale":"2024-05-26T20:19:23.289211+0000","last_undegraded":"2024-05-26T20:19:23.289211+0000","last_fullsized":"2024-05-26T20:19:23.289211+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:56:14.324086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-26T20:19:23.288973+0000","last_change":"2024-05-26T20:19:23.281801+0000","last_active":"2024-05-26T20:19:23.288973+0000","last_peered":"2024-05-26T20:19:23.288973+0000","last_clean":"2024-05-26T20:19:23.288973+0000","last_became_active":"2024-05-26T20:19:23.281578+0000","last_became_peered":"2024-05-26T20:19:23.281578+0000","last_unstale":"2024-05-26T20:19:23.288973+0000","last_undegraded":"2024-05-26T20:19:23.288973+0000","last_fullsized":"2024-05-26T20:19:23.288973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T20:19:22.269160+0000","last_clean_scrub_stamp":"2024-05-26T20:19:22.269160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:53:58.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T20:19:27.093 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-05-26T20:19:27.093 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-05-26T20:19:27.093 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-05-26T20:19:27.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-05-26T20:19:27.458 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-05-26T20:19:27.458 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-26T20:19:27.471 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-05-26T20:19:27.472 INFO:teuthology.run_tasks:Running task rgw... 2024-05-26T20:19:27.487 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-05-26T20:19:27.487 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-05-26T20:19:27.488 INFO:tasks.rgw:Creating data pools 2024-05-26T20:19:27.488 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-05-26T20:19:27.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-05-26T20:19:28.031 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'default.rgw.buckets.data' created 2024-05-26T20:19:28.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-05-26T20:19:29.057 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-05-26T20:19:29.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-05-26T20:19:30.039 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'default.rgw.buckets.index' created 2024-05-26T20:19:30.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-05-26T20:19:31.066 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-05-26T20:19:31.077 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-05-26T20:19:31.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-05-26T20:19:31.425 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'default.rgw.buckets.data' already exists 2024-05-26T20:19:31.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-05-26T20:19:32.040 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-05-26T20:19:32.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-05-26T20:19:32.396 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'default.rgw.buckets.index' already exists 2024-05-26T20:19:32.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-05-26T20:19:33.043 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-05-26T20:19:33.055 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-05-26T20:19:33.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-05-26T20:19:34.294 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' created 2024-05-26T20:19:34.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-05-26T20:19:35.302 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-05-26T20:19:35.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-05-26T20:19:36.306 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' created 2024-05-26T20:19:36.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-05-26T20:19:37.321 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-05-26T20:19:37.334 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-05-26T20:19:37.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-05-26T20:19:37.661 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' already exists 2024-05-26T20:19:37.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-05-26T20:19:38.306 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-05-26T20:19:38.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-05-26T20:19:38.649 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' already exists 2024-05-26T20:19:38.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-05-26T20:19:39.306 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-05-26T20:19:39.318 DEBUG:tasks.rgw:Pools created 2024-05-26T20:19:39.318 INFO:tasks.rgw:Configuring compression type = random 2024-05-26T20:19:39.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-05-26T20:19:39.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-05-26T20:19:39.318 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-05-26T20:19:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:39.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:39.437+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.071+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.071+0000 7f36ea6e9a80 20 realm 2024-05-26T20:19:41.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.071+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.072+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.072+0000 7f36ea6e9a80 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-05-26T20:19:41.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.072+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.072+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.072+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 10 Creating default zonegroup 2024-05-26T20:19:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.073+0000 7f36ea6e9a80 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-05-26T20:19:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.083+0000 7f36ea6e9a80 10 WARNING: driver->list_zones() returned r=-2 2024-05-26T20:19:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.083+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.083+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.085+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.086+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.086+0000 7f36ea6e9a80 10 could not read realm id: (2) No such file or directory 2024-05-26T20:19:41.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.086+0000 7f36ea6e9a80 10 WARNING: failed to set zone as default, r=-22 2024-05-26T20:19:41.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.086+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.086+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.088+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.088+0000 7f36ea6e9a80 20 rados_obj.operate() r=0 bl.length=874 2024-05-26T20:19:41.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.088+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.089+0000 7f36ea6e9a80 20 rados_obj.operate() r=0 bl.length=420 2024-05-26T20:19:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.089+0000 7f36ea6e9a80 10 Using default name default 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.089+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.089+0000 7f36ea6e9a80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.089+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 rados_obj.operate() r=0 bl.length=874 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 zone default found 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 4 Realm: () 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 4 ZoneGroup: default (cf09ab14-e202-460e-a4d0-60d672d7b049) 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 4 Zone: default (fda73892-580d-4109-8ce5-44aab79cb2a7) 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 10 cannot find current period zonegroup using local zonegroup configuration 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 zonegroup default 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:41.090+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:43.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:43.072+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:43.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:43.072+0000 7f36ea6e9a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:43.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:43.074+0000 7f36ea6e9a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:43.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:43.075+0000 7f36ea6e9a80 20 started sync module instance, tier type = 2024-05-26T20:19:43.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:43.075+0000 7f36ea6e9a80 20 started zone id=fda73892-580d-4109-8ce5-44aab79cb2a7 (name=default) with tier type = 2024-05-26T20:19:45.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=0 2024-05-26T20:19:45.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=1 2024-05-26T20:19:45.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=2 2024-05-26T20:19:45.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=3 2024-05-26T20:19:45.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=4 2024-05-26T20:19:45.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=5 2024-05-26T20:19:45.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.096+0000 7f36ea6e9a80 20 add_watcher() i=6 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.097+0000 7f36ea6e9a80 20 add_watcher() i=7 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.097+0000 7f36ea6e9a80 2 all 8 watchers are set, enabling cache 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.097+0000 7f36ea6e9a80 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.097+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.097+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-05-26T20:19:45.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.098+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-05-26T20:19:45.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.099+0000 7f36ea6e9a80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:45.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.102+0000 7f36cf661700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:45.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.102+0000 7f36ea6e9a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:45.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:45.102+0000 7f36ea6e9a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:48.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.073+0000 7f36ea6e9a80 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-05-26T20:19:48.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.073+0000 7f36ea6e9a80 5 note: GC not initialized 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.075+0000 7f36ea6e9a80 20 init_complete bucket index max shards: 11 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.075+0000 7f36ea6e9a80 10 rgw notify: Started notification manager with: 1 workers 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.075+0000 7f36c8e54700 20 rgw notify: INFO: next queues processing will happen at: Sun May 26 20:20:18 2024 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.075+0000 7f36c8632700 20 reqs_thread_entry: start 2024-05-26T20:19:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.075+0000 7f36cae58700 20 reqs_thread_entry: start 2024-05-26T20:19:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.096+0000 7f36cf661700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:48.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.098+0000 7f36ea6e9a80 20 remove_watcher() i=0 2024-05-26T20:19:48.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.098+0000 7f36ea6e9a80 2 removed watcher, disabling cache 2024-05-26T20:19:48.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.099+0000 7f36ea6e9a80 20 remove_watcher() i=1 2024-05-26T20:19:48.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.100+0000 7f36ea6e9a80 20 remove_watcher() i=2 2024-05-26T20:19:48.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.101+0000 7f36ea6e9a80 20 remove_watcher() i=3 2024-05-26T20:19:48.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.102+0000 7f36ea6e9a80 20 remove_watcher() i=4 2024-05-26T20:19:48.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.103+0000 7f36ea6e9a80 20 remove_watcher() i=5 2024-05-26T20:19:48.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.104+0000 7f36ea6e9a80 20 remove_watcher() i=6 2024-05-26T20:19:48.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.105+0000 7f36ea6e9a80 20 remove_watcher() i=7 2024-05-26T20:19:48.108 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-05-26T20:19:48.110 DEBUG:tasks.util.rgw: json result: [] 2024-05-26T20:19:48.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:48.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:48.111 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-05-26T20:19:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.176+0000 7f04539f0a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.176+0000 7f04539f0a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:48.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.180+0000 7f0435161700 20 reqs_thread_entry: start 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "fda73892-580d-4109-8ce5-44aab79cb2a7", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "default.rgw.meta:root", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "default.rgw.control", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "default.rgw.log:gc", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "default.rgw.log:lc", 2024-05-26T20:19:48.186 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "default.rgw.log", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "default.rgw.meta:roles", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "default.rgw.otp", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2024-05-26T20:19:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "default.rgw.buckets.index", 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "default.rgw.buckets.data", 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "compression_type": "random" 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-05-26T20:19:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "", 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "default.rgw.log:notif" 2024-05-26T20:19:48.189 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:19:48.190 DEBUG:tasks.util.rgw: json result: {'id': 'fda73892-580d-4109-8ce5-44aab79cb2a7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-05-26T20:19:48.190 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-05-26T20:19:48.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-05-26T20:19:48.190 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-05-26T20:19:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.245+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.245+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.245+0000 7f5c650bda80 20 realm 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.245+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.246+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.247+0000 7f5c650bda80 20 rados_obj.operate() r=0 bl.length=890 2024-05-26T20:19:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.247+0000 7f5c650bda80 20 searching for the correct realm 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 RGWRados::pool_iterate: got zonegroup_info.cf09ab14-e202-460e-a4d0-60d672d7b049 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 RGWRados::pool_iterate: got zone_info.fda73892-580d-4109-8ce5-44aab79cb2a7 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 RGWRados::pool_iterate: got zone_names.default 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.254+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados_obj.operate() r=0 bl.length=420 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 zone default found 2024-05-26T20:19:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 4 Realm: () 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 4 ZoneGroup: default (cf09ab14-e202-460e-a4d0-60d672d7b049) 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 4 Zone: default (fda73892-580d-4109-8ce5-44aab79cb2a7) 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 10 cannot find current period zonegroup using local zonegroup configuration 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 zonegroup default 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados->read ofs=0 len=0 2024-05-26T20:19:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 started sync module instance, tier type = 2024-05-26T20:19:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.255+0000 7f5c650bda80 20 started zone id=fda73892-580d-4109-8ce5-44aab79cb2a7 (name=default) with tier type = 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=0 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=1 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=2 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=3 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=4 2024-05-26T20:19:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=5 2024-05-26T20:19:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.263+0000 7f5c650bda80 20 add_watcher() i=6 2024-05-26T20:19:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.264+0000 7f5c650bda80 20 add_watcher() i=7 2024-05-26T20:19:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.264+0000 7f5c650bda80 2 all 8 watchers are set, enabling cache 2024-05-26T20:19:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.265+0000 7f5c4a035700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.265+0000 7f5c650bda80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.265+0000 7f5c650bda80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:48.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.269+0000 7f5c650bda80 5 note: GC not initialized 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.270+0000 7f5c4582c700 20 reqs_thread_entry: start 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.270+0000 7f5c650bda80 20 init_complete bucket index max shards: 11 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.270+0000 7f5c650bda80 10 rgw notify: Started notification manager with: 1 workers 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.270+0000 7f5c43828700 20 rgw notify: INFO: next queues processing will happen at: Sun May 26 20:20:18 2024 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-05-26T20:19:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.270+0000 7f5c43006700 20 reqs_thread_entry: start 2024-05-26T20:19:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.278+0000 7f5c4a035700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.279+0000 7f5c650bda80 20 remove_watcher() i=0 2024-05-26T20:19:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.279+0000 7f5c650bda80 2 removed watcher, disabling cache 2024-05-26T20:19:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.280+0000 7f5c650bda80 20 remove_watcher() i=1 2024-05-26T20:19:48.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.281+0000 7f5c650bda80 20 remove_watcher() i=2 2024-05-26T20:19:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.282+0000 7f5c650bda80 20 remove_watcher() i=3 2024-05-26T20:19:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.283+0000 7f5c650bda80 20 remove_watcher() i=4 2024-05-26T20:19:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.284+0000 7f5c650bda80 20 remove_watcher() i=5 2024-05-26T20:19:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.284+0000 7f5c650bda80 20 remove_watcher() i=6 2024-05-26T20:19:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.285+0000 7f5c650bda80 20 remove_watcher() i=7 2024-05-26T20:19:48.287 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-05-26T20:19:48.289 DEBUG:tasks.util.rgw: json result: [] 2024-05-26T20:19:48.289 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:48.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:48.290 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-05-26T20:19:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:48.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.350+0000 7fdab8d41a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:48.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.350+0000 7fdab8d41a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:48.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:19:48.354+0000 7fda9a4b2700 20 reqs_thread_entry: start 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "fda73892-580d-4109-8ce5-44aab79cb2a7", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "default.rgw.meta:root", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "default.rgw.control", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "default.rgw.log:gc", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "default.rgw.log:lc", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "default.rgw.log", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "default.rgw.meta:roles", 2024-05-26T20:19:48.360 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "default.rgw.otp", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:19:48.361 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "default.rgw.buckets.index", 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "default.rgw.buckets.data", 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "compression_type": "random" 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:19:48.362 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:19:48.363 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "", 2024-05-26T20:19:48.363 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "default.rgw.log:notif" 2024-05-26T20:19:48.363 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:19:48.364 DEBUG:tasks.util.rgw: json result: {'id': 'fda73892-580d-4109-8ce5-44aab79cb2a7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-05-26T20:19:48.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-05-26T20:19:48.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-05-26T20:19:48.364 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-05-26T20:19:48.478 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:48.479 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:48.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:48.489+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.341 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.340+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.341 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.340+0000 7fd12885ea80 20 realm 2024-05-26T20:19:50.341 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.340+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.342 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.342+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.342 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.342+0000 7fd12885ea80 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-05-26T20:19:50.342 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.342+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.346 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.346+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.346 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.346+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.347 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.347+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.347 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.347+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.348 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.348+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.348 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.348+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.348+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.348+0000 7fd12885ea80 10 Creating default zonegroup 2024-05-26T20:19:50.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.349+0000 7fd12885ea80 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-05-26T20:19:50.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.358+0000 7fd12885ea80 10 WARNING: driver->list_zones() returned r=-2 2024-05-26T20:19:50.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.358+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.359 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.359+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 10 could not read realm id: (2) No such file or directory 2024-05-26T20:19:50.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 10 WARNING: failed to set zone as default, r=-22 2024-05-26T20:19:50.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.362 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.361+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 20 rados_obj.operate() r=0 bl.length=874 2024-05-26T20:19:50.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 20 rados_obj.operate() r=0 bl.length=420 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 10 Using default name default 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.364+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados_obj.operate() r=0 bl.length=874 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 zone default found 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 4 Realm: () 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 4 ZoneGroup: default (6158b865-3698-4682-941e-fdfa6414cd62) 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 4 Zone: default (c060a9ee-4019-4d94-b559-fda98025852d) 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 10 cannot find current period zonegroup using local zonegroup configuration 2024-05-26T20:19:50.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 zonegroup default 2024-05-26T20:19:50.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:50.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:50.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:50.365+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:52.357 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:52.357+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:52.357+0000 7fd12885ea80 20 rados->read ofs=0 len=0 2024-05-26T20:19:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:52.357+0000 7fd12885ea80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:52.357+0000 7fd12885ea80 20 started sync module instance, tier type = 2024-05-26T20:19:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:52.357+0000 7fd12885ea80 20 started zone id=c060a9ee-4019-4d94-b559-fda98025852d (name=default) with tier type = 2024-05-26T20:19:54.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=0 2024-05-26T20:19:54.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=1 2024-05-26T20:19:54.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=2 2024-05-26T20:19:54.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=3 2024-05-26T20:19:54.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=4 2024-05-26T20:19:54.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=5 2024-05-26T20:19:54.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.387+0000 7fd12885ea80 20 add_watcher() i=6 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 add_watcher() i=7 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 2 all 8 watchers are set, enabling cache 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-05-26T20:19:54.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.388+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.389+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.389+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.389+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.389+0000 7fd12885ea80 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.389+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-05-26T20:19:54.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.390+0000 7fd12885ea80 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-05-26T20:19:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.392+0000 7fd10d7d6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.393+0000 7fd12885ea80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:54.393+0000 7fd12885ea80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.365+0000 7fd12885ea80 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.365+0000 7fd12885ea80 5 note: GC not initialized 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.366+0000 7fd108fcd700 20 reqs_thread_entry: start 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.366+0000 7fd12885ea80 20 init_complete bucket index max shards: 11 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.366+0000 7fd12885ea80 10 rgw notify: Started notification manager with: 1 workers 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.366+0000 7fd106fc9700 20 rgw notify: INFO: next queues processing will happen at: Sun May 26 20:20:27 2024 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-05-26T20:19:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.367+0000 7fd1067a7700 20 reqs_thread_entry: start 2024-05-26T20:19:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.385+0000 7fd10d7d6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.387+0000 7fd12885ea80 20 remove_watcher() i=0 2024-05-26T20:19:57.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.387+0000 7fd12885ea80 2 removed watcher, disabling cache 2024-05-26T20:19:57.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.388+0000 7fd12885ea80 20 remove_watcher() i=1 2024-05-26T20:19:57.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.390+0000 7fd12885ea80 20 remove_watcher() i=2 2024-05-26T20:19:57.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.393+0000 7fd12885ea80 20 remove_watcher() i=3 2024-05-26T20:19:57.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.396+0000 7fd12885ea80 20 remove_watcher() i=4 2024-05-26T20:19:57.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.398+0000 7fd12885ea80 20 remove_watcher() i=5 2024-05-26T20:19:57.399 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.399+0000 7fd12885ea80 20 remove_watcher() i=6 2024-05-26T20:19:57.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.401+0000 7fd12885ea80 20 remove_watcher() i=7 2024-05-26T20:19:57.404 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2024-05-26T20:19:57.406 DEBUG:tasks.util.rgw: json result: [] 2024-05-26T20:19:57.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:57.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:57.407 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-05-26T20:19:57.460 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:57.460 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:57.469 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.469+0000 7f29f60e0a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:57.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.469+0000 7f29f60e0a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:57.473 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.473+0000 7f29d7851700 20 reqs_thread_entry: start 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "c060a9ee-4019-4d94-b559-fda98025852d", 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default", 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "default.rgw.meta:root", 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "default.rgw.control", 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "default.rgw.log:gc", 2024-05-26T20:19:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "default.rgw.log:lc", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "default.rgw.log", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "default.rgw.meta:roles", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "default.rgw.otp", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "", 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "" 2024-05-26T20:19:57.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "default.rgw.buckets.index", 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "default.rgw.buckets.data", 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-05-26T20:19:57.484 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "", 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "default.rgw.log:notif" 2024-05-26T20:19:57.485 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:19:57.486 DEBUG:tasks.util.rgw: json result: {'id': 'c060a9ee-4019-4d94-b559-fda98025852d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-05-26T20:19:57.486 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-05-26T20:19:57.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-05-26T20:19:57.486 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-05-26T20:19:57.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:57.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:57.538 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.538+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.539 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.539 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 realm 2024-05-26T20:19:57.539 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.539 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.539+0000 7fba95ad6a80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.540+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.540+0000 7fba95ad6a80 20 rados_obj.operate() r=0 bl.length=890 2024-05-26T20:19:57.541 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.540+0000 7fba95ad6a80 20 searching for the correct realm 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 RGWRados::pool_iterate: got zonegroup_info.6158b865-3698-4682-941e-fdfa6414cd62 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 RGWRados::pool_iterate: got zone_info.c060a9ee-4019-4d94-b559-fda98025852d 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 RGWRados::pool_iterate: got zone_names.default 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados_obj.operate() r=0 bl.length=46 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados_obj.operate() r=0 bl.length=420 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 zone default found 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 4 Realm: () 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 4 ZoneGroup: default (6158b865-3698-4682-941e-fdfa6414cd62) 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 4 Zone: default (c060a9ee-4019-4d94-b559-fda98025852d) 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 10 cannot find current period zonegroup using local zonegroup configuration 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 zonegroup default 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.547+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 rados->read ofs=0 len=0 2024-05-26T20:19:57.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 rados_obj.operate() r=-2 bl.length=0 2024-05-26T20:19:57.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 started sync module instance, tier type = 2024-05-26T20:19:57.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.548+0000 7fba95ad6a80 20 started zone id=c060a9ee-4019-4d94-b559-fda98025852d (name=default) with tier type = 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=0 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=1 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=2 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=3 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=4 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=5 2024-05-26T20:19:57.572 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.572+0000 7fba95ad6a80 20 add_watcher() i=6 2024-05-26T20:19:57.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.576+0000 7fba95ad6a80 20 add_watcher() i=7 2024-05-26T20:19:57.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.576+0000 7fba95ad6a80 2 all 8 watchers are set, enabling cache 2024-05-26T20:19:57.577 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.577+0000 7fba7aa4e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:57.578 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.578+0000 7fba95ad6a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:57.578 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.578+0000 7fba95ad6a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:57.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.581+0000 7fba95ad6a80 5 note: GC not initialized 2024-05-26T20:19:57.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.582+0000 7fba76245700 20 reqs_thread_entry: start 2024-05-26T20:19:57.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.582+0000 7fba95ad6a80 20 init_complete bucket index max shards: 11 2024-05-26T20:19:57.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.582+0000 7fba95ad6a80 10 rgw notify: Started notification manager with: 1 workers 2024-05-26T20:19:57.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.582+0000 7fba74241700 20 rgw notify: INFO: next queues processing will happen at: Sun May 26 20:20:27 2024 2024-05-26T20:19:57.583 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-05-26T20:19:57.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.582+0000 7fba73a1f700 20 reqs_thread_entry: start 2024-05-26T20:19:57.589 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.589+0000 7fba7aa4e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-26T20:19:57.593 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.593+0000 7fba95ad6a80 20 remove_watcher() i=0 2024-05-26T20:19:57.593 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.593+0000 7fba95ad6a80 2 removed watcher, disabling cache 2024-05-26T20:19:57.593 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.593+0000 7fba95ad6a80 20 remove_watcher() i=1 2024-05-26T20:19:57.595 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.595+0000 7fba95ad6a80 20 remove_watcher() i=2 2024-05-26T20:19:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.598+0000 7fba95ad6a80 20 remove_watcher() i=3 2024-05-26T20:19:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.599+0000 7fba95ad6a80 20 remove_watcher() i=4 2024-05-26T20:19:57.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.600+0000 7fba95ad6a80 20 remove_watcher() i=5 2024-05-26T20:19:57.601 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.601+0000 7fba95ad6a80 20 remove_watcher() i=6 2024-05-26T20:19:57.602 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.602+0000 7fba95ad6a80 20 remove_watcher() i=7 2024-05-26T20:19:57.604 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2024-05-26T20:19:57.606 DEBUG:tasks.util.rgw: json result: [] 2024-05-26T20:19:57.606 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:57.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-05-26T20:19:57.607 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-05-26T20:19:57.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:57.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:57.704 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.704+0000 7fdaec2e1a80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-26T20:19:57.704 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.704+0000 7fdaec2e1a80 20 check_secure_mon_conn(): mode 1 is insecure 2024-05-26T20:19:57.707 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:19:57.707+0000 7fdacda52700 20 reqs_thread_entry: start 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "c060a9ee-4019-4d94-b559-fda98025852d", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "default.rgw.meta:root", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "default.rgw.control", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "default.rgw.log:gc", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "default.rgw.log:lc", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "default.rgw.log", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "default.rgw.meta:roles", 2024-05-26T20:19:57.713 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "default.rgw.otp", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "" 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-05-26T20:19:57.714 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "default.rgw.buckets.index", 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "default.rgw.buckets.data", 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:19:57.715 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:19:57.716 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "", 2024-05-26T20:19:57.716 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "default.rgw.log:notif" 2024-05-26T20:19:57.716 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:19:57.717 DEBUG:tasks.util.rgw: json result: {'id': 'c060a9ee-4019-4d94-b559-fda98025852d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-05-26T20:19:57.717 INFO:tasks.rgw:Starting rgw... 2024-05-26T20:19:57.717 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-05-26T20:19:57.717 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-26T20:19:57.717 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-05-26T20:19:57.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-05-26T20:19:57.720 INFO:tasks.rgw.c1.client.0:Started 2024-05-26T20:19:57.720 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-05-26T20:19:57.720 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-26T20:19:57.720 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-05-26T20:19:57.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-05-26T20:19:57.722 INFO:tasks.rgw.c1.client.1:Started 2024-05-26T20:19:57.723 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-05-26T20:19:57.723 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-26T20:19:57.723 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-05-26T20:19:57.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-05-26T20:19:57.725 INFO:tasks.rgw.c2.client.0:Started 2024-05-26T20:19:57.725 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-05-26T20:19:57.725 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-26T20:19:57.725 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-05-26T20:19:57.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-05-26T20:19:57.727 INFO:tasks.rgw.c2.client.1:Started 2024-05-26T20:19:57.727 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi007.front.sepia.ceph.com:8000/ 2024-05-26T20:19:57.728 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8000/ 2024-05-26T20:19:57.784 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:19:57.784 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:19:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T20:19:57.786 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi007.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T20:19:58.787 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8000/ 2024-05-26T20:19:58.814 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:19:58.814 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:19:58.815 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:19:58.816 INFO:teuthology.orchestra.run.smithi007.stdout:anonymous 2024-05-26T20:19:58.816 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi007.front.sepia.ceph.com:8001/ 2024-05-26T20:19:58.817 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8001/ 2024-05-26T20:19:58.883 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:19:58.884 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:19:58.885 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:19:58.885 INFO:teuthology.orchestra.run.smithi007.stdout:anonymous 2024-05-26T20:19:58.885 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8000/ 2024-05-26T20:19:58.885 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000/ 2024-05-26T20:19:58.917 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:19:58.917 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:19:58.918 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:19:58.918 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T20:19:58.918 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8001/ 2024-05-26T20:19:58.918 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001/ 2024-05-26T20:19:58.980 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:19:58.980 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:19:58.980 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:19:58.981 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T20:19:58.981 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-05-26T20:19:59.900 INFO:tasks.rgw_multisite:creating realm.. 2024-05-26T20:19:59.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:19:59.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:19:59.901 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:19:59.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:19:59.956 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:19:59.983 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:19:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:19:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-realm", 2024-05-26T20:19:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "current_period": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T20:19:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1 2024-05-26T20:19:59.984 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:19:59.985 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-05-26T20:19:59.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:19:59.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:19:59.986 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:20:00.051+0000 7f3968974a80 -1 old format 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T20:20:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "", 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [], 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [] 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:00.060 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:00.061 INFO:tasks.rgw_multisite:creating master zone.. 2024-05-26T20:20:00.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi007.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:00.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi007.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:00.062 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key WB0LE5I0BAFHL536 --secret TUtFQih6d8vJyiEWzurVtHMAn8IwKknK --endpoints http://smithi007.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:00.174 INFO:teuthology.orchestra.run.smithi007.stderr:NOTICE: set zone's realm_id=a03a6f99-1643-481f-8ae9-2762eec2e396 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "test-zone1.rgw.control", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "test-zone1.rgw.log", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-05-26T20:20:00.188 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:20:00.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:00.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:00.191 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:00.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:00.191 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-05-26T20:20:00.191 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:00.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-05-26T20:20:01.101 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-05-26T20:20:01.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-05-26T20:20:02.127 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-05-26T20:20:02.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:02.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:02.144 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:02.200 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:02.219 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:02.219 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "test-zone1.rgw.control", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "test-zone1.rgw.log", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-05-26T20:20:02.220 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:20:02.221 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "compression_type": "random" 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-05-26T20:20:02.222 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:02.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:02.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:02.224 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:02.273 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:20:02.286+0000 7f6b30583a80 0 period (00ea8d6c-6a72-476f-b08d-299358d5f544 does not have zone 7e72b8a9-5d69-4e33-98d3-490830e32745 configured 2024-05-26T20:20:09.192 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:09.192 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1, 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T20:20:09.193 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T20:20:09.194 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": false, 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:09.195 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:09.196 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:09.197 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T20:20:09.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:09.199 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:09.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:09.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:09.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:09.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T20:20:09.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:20:09.203 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:09.204 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-05-26T20:20:09.204 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T20:20:09.205 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T20:20:09.205+0000 7f1467038700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 47466) UID: 0 2024-05-26T20:20:09.205 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T20:20:09.205+0000 7f146b2a5a80 -1 shutting down 2024-05-26T20:20:15.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-05-26T20:20:15.306 INFO:tasks.rgw.c1.client.0:Stopped 2024-05-26T20:20:15.306 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-05-26T20:20:15.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-05-26T20:20:15.309 INFO:tasks.rgw.c1.client.0:Started 2024-05-26T20:20:15.309 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8000 2024-05-26T20:20:15.336 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:15.336 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T20:20:15.337 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi007.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T20:20:16.338 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8000 2024-05-26T20:20:16.363 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:16.363 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:16.365 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:20:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:anonymous 2024-05-26T20:20:16.366 INFO:tasks.rgw_multisite:creating admin user.. 2024-05-26T20:20:16.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.366 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key WB0LE5I0BAFHL536 --secret TUtFQih6d8vJyiEWzurVtHMAn8IwKknK --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:16.503 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "user_id": "realm-admin", 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "display_name": "Realm Admin", 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "email": "", 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "suspended": 0, 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "max_buckets": 1000, 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "subusers": [], 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "keys": [ 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "realm-admin", 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:16.504 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "swift_keys": [], 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": [], 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "op_mask": "read, write, delete", 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "system": true, 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "", 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_storage_class": "", 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_tags": [], 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:16.505 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "temp_url_keys": [], 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "rgw", 2024-05-26T20:20:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: "mfa_ids": [] 2024-05-26T20:20:16.507 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:16.507 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:20:16.516 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-05-26T20:20:16.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi007.front.sepia.ceph.com:8000', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi007.front.sepia.ceph.com:8000', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.517 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi007.front.sepia.ceph.com:8000 --access-key WB0LE5I0BAFHL536 --secret TUtFQih6d8vJyiEWzurVtHMAn8IwKknK --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:16.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:16.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:16.613 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:16.613 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:16.614 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-realm", 2024-05-26T20:20:16.614 INFO:teuthology.orchestra.run.smithi137.stdout: "current_period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:16.614 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 2 2024-05-26T20:20:16.614 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:16.615 INFO:tasks.rgw_multisite:creating zone.. 2024-05-26T20:20:16.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:16.616 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key WB0LE5I0BAFHL536 --secret TUtFQih6d8vJyiEWzurVtHMAn8IwKknK --endpoints http://smithi137.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:16.670 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:16.670 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:16.686 INFO:teuthology.orchestra.run.smithi137.stderr:NOTICE: set zone's realm_id=a03a6f99-1643-481f-8ae9-2762eec2e396 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone2", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "test-zone2.rgw.control", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "test-zone2.rgw.log", 2024-05-26T20:20:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-05-26T20:20:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-05-26T20:20:16.703 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:16.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-05-26T20:20:17.395 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-05-26T20:20:17.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-05-26T20:20:18.430 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-05-26T20:20:18.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:18.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:18.450 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:18.503 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:18.503 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:18.522 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone2", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "test-zone2.rgw.control", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "test-zone2.rgw.log", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-05-26T20:20:18.523 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:18.524 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:18.525 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:18.526 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:18.526 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-05-26T20:20:18.526 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:18.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:18.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:18.527 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:18.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:18.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:18.587 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:18.588+0000 7f48cff86a80 0 period (de3b77f2-60b7-47e4-b23e-f2e513d5e591 does not have zone 5d9078f7-c6b6-41b6-8e8a-a69458435742 configured 2024-05-26T20:20:20.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-05-26T20:20:25.453 INFO:teuthology.orchestra.run.smithi137.stderr:Sending period to new master zone 7e72b8a9-5d69-4e33-98d3-490830e32745 2024-05-26T20:20:25.476 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 2, 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": [], 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "period_map": { 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "zonegroups": [ 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "test-zonegroup", 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zonegroup", 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": true, 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:20:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone2", 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone1", 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-05-26T20:20:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-05-26T20:20:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:20:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "short_zone_ids": [ 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3171627794 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3772860827 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "period_config": { 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_quota": { 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "user_quota": { 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-05-26T20:20:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "user_ratelimit": { 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_ratelimit": { 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:20:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "anonymous_ratelimit": { 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:20:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_name": "test-realm", 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:25.491 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:25.492 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-05-26T20:20:25.492 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T20:20:25.493 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T20:20:25.494+0000 7f8a7fd3f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 47358) UID: 0 2024-05-26T20:20:25.493 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T20:20:25.494+0000 7f8a83faca80 -1 shutting down 2024-05-26T20:20:25.592 INFO:tasks.rgw.c2.client.0:Stopped 2024-05-26T20:20:25.592 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-05-26T20:20:25.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-05-26T20:20:25.634 INFO:tasks.rgw.c2.client.0:Started 2024-05-26T20:20:25.635 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T20:20:25.655 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:25.656 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T20:20:25.658 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T20:20:26.659 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T20:20:26.680 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:26.681 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:26.682 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:20:26.682 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T20:20:26.683 INFO:tasks.rgw_multisite:creating zone.. 2024-05-26T20:20:26.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi007.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:26.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'WB0LE5I0BAFHL536', '--secret', 'TUtFQih6d8vJyiEWzurVtHMAn8IwKknK', '--endpoints', 'http://smithi007.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:26.683 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key WB0LE5I0BAFHL536 --secret TUtFQih6d8vJyiEWzurVtHMAn8IwKknK --endpoints http://smithi007.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:26.746 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:26.746 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:26.764 INFO:teuthology.orchestra.run.smithi007.stderr:NOTICE: set zone's realm_id=a03a6f99-1643-481f-8ae9-2762eec2e396 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "test-zone3.rgw.control", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "test-zone3.rgw.log", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-05-26T20:20:26.780 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:20:26.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:26.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:26.783 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:26.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:26.783 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-05-26T20:20:26.783 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:26.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-05-26T20:20:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-05-26T20:20:27.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-05-26T20:20:28.198 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-05-26T20:20:28.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:28.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:28.211 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:28.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:28.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "test-zone3.rgw.control", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "test-zone3.rgw.log", 2024-05-26T20:20:28.291 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:28.292 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "compression_type": "random" 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T20:20:28.293 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-05-26T20:20:28.294 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:28.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:28.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:28.296 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:28.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:20:28.357+0000 7f2c3e003a80 0 period (de3b77f2-60b7-47e4-b23e-f2e513d5e591 does not have zone 2be55b7a-f70c-4aa7-a1f4-aea3d2de43be configured 2024-05-26T20:20:35.251 INFO:teuthology.orchestra.run.smithi007.stderr:Sending period to new master zone 7e72b8a9-5d69-4e33-98d3-490830e32745 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 3, 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:35.273 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T20:20:35.274 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:35.276 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T20:20:35.277 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T20:20:35.278 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T20:20:35.279 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T20:20:35.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.281 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T20:20:35.281 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T20:20:35.281 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:20:35.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:35.283 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T20:20:35.284 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.285 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T20:20:35.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:35.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:35.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:20:35.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T20:20:35.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:20:35.287 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:35.288 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-05-26T20:20:35.288 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T20:20:35.289 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T20:20:35.289+0000 7fdaec880700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 47467) UID: 0 2024-05-26T20:20:35.289 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T20:20:35.289+0000 7fdaf0aeda80 -1 shutting down 2024-05-26T20:20:37.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-05-26T20:20:41.390 INFO:tasks.rgw.c1.client.1:Stopped 2024-05-26T20:20:41.391 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-05-26T20:20:41.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-05-26T20:20:41.394 INFO:tasks.rgw.c1.client.1:Started 2024-05-26T20:20:41.394 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8001 2024-05-26T20:20:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T20:20:41.423 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:41.423 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:41.423 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi007.front.sepia.ceph.com port 8001: Connection refused 2024-05-26T20:20:42.424 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8001 2024-05-26T20:20:42.448 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T20:20:42.448 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T20:20:42.449 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T20:20:42.450 INFO:teuthology.orchestra.run.smithi007.stdout:anonymous 2024-05-26T20:20:42.450 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-05-26T20:20:42.450 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-05-26T20:20:42.466 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi 2024-05-26T20:20:42.529 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-05-26T20:20:42.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:42.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:20:42.530 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T20:20:42.589 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:42.590 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:42.624 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "user_id": "rgw-multisite-test-user", 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "display_name": "Multisite Test User", 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "email": "", 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "suspended": 0, 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "max_buckets": 1000, 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "subusers": [], 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "keys": [ 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "rgw-multisite-test-user", 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "SVLBD58NLW55IW1K7YLC", 2024-05-26T20:20:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "S7HDB8fTRhZJxclKOgOZP0DA8wfk5FBzc13MIs6X" 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "swift_keys": [], 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": [ 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "roles", 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "perm": "*" 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "op_mask": "read, write, delete", 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "", 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "default_storage_class": "", 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_tags": [], 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "temp_url_keys": [], 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "rgw", 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "mfa_ids": [] 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:20:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-05-26T20:20:42.636 INFO:rgw_multi.tests:meta checkpoint 2024-05-26T20:20:42.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:42.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:42.638 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:20:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:20:42.728 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:20:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:20:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:20:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:20:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:20:42.730 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:20:42.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:20:42.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:42.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:42.731 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:20:42.788 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:42.788 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:42.818 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:42.818+0000 7fc4275b1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:42.819 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:42.819+0000 7fc4275b1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:42.831 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:20:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:20:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:20:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:42.833 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:42.834 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:42.835 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:20:42Z" 2024-05-26T20:20:42.836 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:42.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:20:43.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-05-26T20:20:47.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:47.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:47.839 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:20:47.896 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:47.896 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:47.926 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:47.925+0000 7fefbe1d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:47.926 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:47.926+0000 7fefbe1d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.938 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:20:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:20:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:20:47Z" 2024-05-26T20:20:47.943 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:47.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:20:52.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:52.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:52.945 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:20:53.002 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:53.002 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:53.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:53.034+0000 7fda074bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:53.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:53.035+0000 7fda074bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:53.047 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:53.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:53.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:53.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:20:53Z" 2024-05-26T20:20:53.052 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:53.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:20:58.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:58.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:20:58.055 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:20:58.113 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:20:58.113 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:20:58.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:58.143+0000 7f9f4763ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:58.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:20:58.144+0000 7f9f4763ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:20:58.155 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:20:58.156 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:20:58.157 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:20:58.158 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:20:58Z" 2024-05-26T20:20:58.161 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:20:58.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:03.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:03.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:03.164 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:03.220 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:03.221 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:03.250 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:03.250+0000 7f63258d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:03.251 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:03.251+0000 7f63258d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:03.263 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:03.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:03.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:03.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:03.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:03Z" 2024-05-26T20:21:03.268 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:03.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:08.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:08.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:08.270 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:08.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:08.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:08.360 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:08.361+0000 7f2da5c83a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:08.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:08.362+0000 7f2da5c83a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:08Z" 2024-05-26T20:21:08.378 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:08.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:13.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:13.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:13.381 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:13.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:13.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:13.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:13.469+0000 7f2949243a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:13.469 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:13.470+0000 7f2949243a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:13.481 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:13.488 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:13Z" 2024-05-26T20:21:13.489 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:13.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:18.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:18.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:18.492 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:18.579 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:18.580+0000 7f9ff37b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:18.580 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:18.581+0000 7f9ff37b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.592 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:18.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:18.594 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:18.595 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:18.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:18Z" 2024-05-26T20:21:18.597 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:18.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:23.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:23.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:23.599 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:23.656 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:23.656 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:23.689 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:23.689+0000 7f1da9032a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:23.690 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:23.690+0000 7f1da9032a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:23.701 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:23.701 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.702 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:23.703 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:23.704 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:23.705 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:23.706 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:23.707 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:23.707 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:23.707 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:23.707 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:23.707 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:23.708 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:23.708 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:23.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:23.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:23.709 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:23.709 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:23.709 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:23.709 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:23.709 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:23.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:23.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:23.712 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:23Z" 2024-05-26T20:21:23.712 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:23.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:28.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:28.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:28.715 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:28.801 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:28.801+0000 7ff984affa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:28.802 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:28.802+0000 7ff984affa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:28.817 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:28.818 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:28Z" 2024-05-26T20:21:28.819 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:28.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:33.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:33.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:33.821 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:33.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:33.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:33.908+0000 7f8e33dbfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:33.909+0000 7f8e33dbfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:33Z" 2024-05-26T20:21:33.926 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:33.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:38.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:38.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:38.929 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:38.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:38.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:39.017 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:39.017+0000 7f89e1eaba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:39.018 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:39.018+0000 7f89e1eaba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:39.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:39Z" 2024-05-26T20:21:39.035 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:39.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:44.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:44.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:44.038 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:44.094 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:44.094 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:44.124 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:44.124+0000 7f28f3d5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:44.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:44.125+0000 7f28f3d5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:44.136 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:44.137 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:44.138 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:44.139 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:44.140 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:44.141 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:44.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:44.142 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:44Z" 2024-05-26T20:21:44.142 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:44.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:49.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:49.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:49.144 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:49.201 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:49.201 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:49.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:49.234+0000 7f9544b06a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:49.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:49.235+0000 7f9544b06a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:49.246 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:49.246 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:49.246 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:49.246 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:49.246 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:49.249 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:49.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:49Z" 2024-05-26T20:21:49.251 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:49.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:54.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:54.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:54.254 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:54.310 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:54.311 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:54.343 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:54.342+0000 7fec689bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:54.343 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:54.343+0000 7fec689bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:54.355 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:54.355 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:54.356 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:54.357 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:54.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:54Z" 2024-05-26T20:21:54.360 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:54.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:21:59.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:59.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:21:59.363 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:21:59.419 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:21:59.420 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:21:59.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:59.450+0000 7fc7ce323a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:59.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:21:59.451+0000 7fc7ce323a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:21:59.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:59.464 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:59.465 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:21:59.466 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:21:59.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:21:59.468 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:21:59Z" 2024-05-26T20:21:59.468 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:21:59.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:04.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:04.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:04.470 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:04.527 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:04.528 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:04.557 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:04.558+0000 7fda44bc5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:04.558 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:04.559+0000 7fda44bc5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:04.569 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:04.569 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:04.570 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:04.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:04.572 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:04.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:04Z" 2024-05-26T20:22:04.574 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:04.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:09.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:09.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:09.577 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:09.632 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:09.632 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:09.662 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:09.663+0000 7f5315fe0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:09.663 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:09.664+0000 7f5315fe0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:09Z" 2024-05-26T20:22:09.680 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:09.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:14.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:14.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:14.682 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:14.739 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:14.740 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:14.773 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:14.774+0000 7fa5005fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:14.774 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:14.775+0000 7fa5005fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:14.785 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:14.786 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:14.788 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:14.789 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:14Z" 2024-05-26T20:22:14.790 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:14.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:19.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:19.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:19.793 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:19.850 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:19.851 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:19.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:19.881+0000 7fc2130afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:19.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:19.882+0000 7fc2130afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:19.893 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:19Z" 2024-05-26T20:22:19.904 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:19.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:24.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:24.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:24.907 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:24.964 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:24.964 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:24.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:24.994+0000 7f76d19b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:24.995 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:24.995+0000 7f76d19b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:25.006 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:24Z" 2024-05-26T20:22:25.011 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:25.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:30.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:30.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:30.013 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:30.073 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:30.074 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:30.103 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:30.102+0000 7f1c4f3bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:30.103 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:30.103+0000 7f1c4f3bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:30.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:30.116 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:30.117 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:30.118 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:30.119 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:30.120 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:30.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:30.120 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:30Z" 2024-05-26T20:22:30.120 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:30.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:35.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:35.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:35.123 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:35.210 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:35.210+0000 7f5953c8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:35.211 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:35.211+0000 7f5953c8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:35.223 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:35.224 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:35.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.224 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:35.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:35.225 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:35.226 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:35.227 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:35.228 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:35.229 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:35.229 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:35.229 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:35.229 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:35.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:35.231 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:35Z" 2024-05-26T20:22:35.231 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:35.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1716754842.623606_31.1 target= 2024-05-26T20:22:40.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.233 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:40.289 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:40.290 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:40.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:40.319+0000 7f009e185a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.320 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:40.320+0000 7f009e185a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:40Z" 2024-05-26T20:22:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:40.338 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:40.338 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:40.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.338 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:40.397 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:40.397 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:40.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:40.429+0000 7f957c87fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:40.430+0000 7f957c87fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.441 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:40.441 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:40.441 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:40.441 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:40.442 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.443 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.444 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.445 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.446 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:40Z" 2024-05-26T20:22:40.447 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:40.448 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:40.448 INFO:rgw_multi.tests:meta checkpoint 2024-05-26T20:22:40.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.449 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:40.499 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:40.500 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:40.530 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:40.540 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:40.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.542 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:40.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.542 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:40.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:40.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.543 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:40.594 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:40.595 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:40.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:40.624+0000 7ff35b53da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:40.625+0000 7ff35b53da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.635 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:40.635 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.636 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:40.637 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.639 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:40.640 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:40Z" 2024-05-26T20:22:40.641 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:40.643 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:40.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:40.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:40.644 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:40.695 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:40.695 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:40.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:40.722+0000 7fb0806f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:40.723+0000 7fb0806f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:40Z" 2024-05-26T20:22:40.739 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:40.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:40.740 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-05-26T20:22:41.179 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-1 2024-05-26T20:22:43.338 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-2 2024-05-26T20:22:45.645 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-3 2024-05-26T20:22:45.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:45.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:45.675 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:45.731 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:45.731 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:45.766 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754965.665091_46.1", 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:45.665091Z" 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:45.779 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:45.780 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:45.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:45.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:45.780 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:45.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:45.859+0000 7fc86f797a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:45.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:45.860+0000 7fc86f797a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:45.871 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:45.874 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754963.313150_35.1", 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:45.875 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:43.313150Z", 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:45Z" 2024-05-26T20:22:45.876 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:45.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1716754965.665091_46.1 target=1_1716754963.313150_35.1 2024-05-26T20:22:50.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:50.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:50.879 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:50.936 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:50.936 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:50.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:50.967+0000 7f6203ccba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:50.968 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:50.968+0000 7f6203ccba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.980 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754965.665091_46.1", 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:45.665091Z", 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:50.984 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:50Z" 2024-05-26T20:22:50.985 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:50.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:50.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:50.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:50.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:50.987 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.082+0000 7f1e4b03ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.084+0000 7f1e4b03ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.097 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:51.099 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.100 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754965.665091_46.1", 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:45.665091Z", 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.101 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:51Z" 2024-05-26T20:22:51.102 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:51.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:51.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-05-26T20:22:51.122 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-4 2024-05-26T20:22:51.134 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-5 2024-05-26T20:22:51.155 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-6 2024-05-26T20:22:51.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.176 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:51.168478Z" 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.269 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:51.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:51.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.270 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:51.372+0000 7ff5c5f47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.373 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:51.373+0000 7ff5c5f47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:51.384 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:51.385 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:51.168478Z", 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:51.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:51Z" 2024-05-26T20:22:51.389 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:51.390 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:51.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:51.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.390 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.468+0000 7f2d21b4fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.469+0000 7f2d21b4fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.479 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:51.480 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.481 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.482 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:51.483 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:51.168478Z", 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:51Z" 2024-05-26T20:22:51.484 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:51.485 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:51.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.719 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.769 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.769 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:51.168478Z" 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:51.810 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:51.811 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:51.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.812 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.863 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.863 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:51.892+0000 7f4a7c3c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:51.893+0000 7f4a7c3c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:51.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.906 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:51.168478Z", 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:51.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:51.909 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:51Z" 2024-05-26T20:22:51.909 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:51.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:51.910 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:51.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:51.910 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.990+0000 7f508ebaaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:51.991+0000 7f508ebaaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:52.003 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.004 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.005 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.006 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754971.168478_58.1", 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:51.168478Z", 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:52.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:51Z" 2024-05-26T20:22:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:52.009 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:52.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-05-26T20:22:52.028 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-7 2024-05-26T20:22:52.039 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-8 2024-05-26T20:22:52.064 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-9 2024-05-26T20:22:52.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.084 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.077078_70.1", 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.077078Z" 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.185 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:52.186 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:52.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.187 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:52.236 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:52.236 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:52.265+0000 7f30bbdeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:52.266+0000 7f30bbdeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.276 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.278 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.280 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.077078_70.1", 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.077078Z", 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:52.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:52.282 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:52.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.282 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:52Z" 2024-05-26T20:22:52.282 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:52.283 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:52.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:52.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.283 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:52.362+0000 7f365e2c7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:52.363+0000 7f365e2c7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.373 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.077078_70.1", 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.077078Z", 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:52Z" 2024-05-26T20:22:52.383 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:52.384 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:52.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.533 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:52.625 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:52.626 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:52.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:52.627 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:52.676 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:52.676 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:52.712 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:52.712+0000 7f78c1c77a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.713 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:52.713+0000 7f78c1c77a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:52.724 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:52.724 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:52.724 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:52.724 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.725 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.726 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.412642_71.1", 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.412642Z", 2024-05-26T20:22:52.729 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:52Z" 2024-05-26T20:22:52.730 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:52.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1716754972.511255_82.1 target=1_1716754972.412642_71.1 2024-05-26T20:22:57.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:57.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:57.733 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:57.790 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:57.790 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:57.820+0000 7fc02086ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:57.821 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:57.821+0000 7fc02086ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:57.833 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:57.834 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:57.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:57.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:57Z" 2024-05-26T20:22:57.838 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:57.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:22:57.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:22:57.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:57.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:57.839 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:57.897 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:57.898 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:57.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:57.929+0000 7f12ec6cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:22:57.930+0000 7f12ec6cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:57.942 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:22:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:22:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:22:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:22:57Z" 2024-05-26T20:22:57.947 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:22:57.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:22:57.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-05-26T20:22:57.957 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-10 2024-05-26T20:22:57.969 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-11 2024-05-26T20:22:57.991 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-12 2024-05-26T20:22:58.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:58.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:58.363 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:58.442 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:22:58.453 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754978.004677_46.1", 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:58.004677Z" 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:22:58.454 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:22:58.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:22:58.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:58.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:22:58.456 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:22:58.508 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:22:58.508 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:22:58.537 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:58.537+0000 7fc4aaee5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:58.538 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:22:58.538+0000 7fc4aaee5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:22:58.549 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:22:58.549 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:58.550 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:58.551 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:58.552 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754977.963911_35.1", 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:57.963911Z", 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:22:58.553 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:22:58.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:22:58.555 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:22:58Z" 2024-05-26T20:22:58.555 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:22:58.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716754978.004677_46.1 target=1_1716754977.963911_35.1 2024-05-26T20:23:03.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:23:03.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:23:03.557 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:23:03.614 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:23:03.614 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:23:03.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:03.646+0000 7fe7ae470a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:23:03.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:03.647+0000 7fe7ae470a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:23:03.660 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:23:03.661 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:23:03.662 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:23:03.663 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754978.004677_46.1", 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:58.004677Z", 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:23:03.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:23:03.665 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:23:03.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:23:03.667 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:23:03Z" 2024-05-26T20:23:03.667 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:23:03.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:23:03.669 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:23:03.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:23:03.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:23:03.669 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:23:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:23:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:23:03.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:03.760+0000 7f154899ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:23:03.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:03.761+0000 7f154899ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:23:03.779 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:23:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:23:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:23:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754978.004677_46.1", 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:58.004677Z", 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:23:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:23:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:23:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:23:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:23:03Z" 2024-05-26T20:23:03.785 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:23:03.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:23:03.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:03.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:03.785 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:23:03.843 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:23:03.844 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:23:03.877 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:03.878+0000 7fc1fa9d5a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:08.878 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:08.879+0000 7fc1fa9d5a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:13.879 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:13.879+0000 7fc1fa9d5a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:18.880 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:18.881+0000 7fc1fa9d5a80 1 bucket sync caught up with source: 2024-05-26T20:23:18.881 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.5.6, , 00000000001.4.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2024-05-26T20:23:18.881 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000001.5.6, , 00000000001.4.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2024-05-26T20:23:18.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:23:18.881+0000 7fc1fa9d5a80 0 bucket checkpoint complete 2024-05-26T20:23:18.896 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-10 zones={test-zone2, test-zone1} 2024-05-26T20:23:18.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:23:18.907 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:23:18.907 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:23:18.907 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:23:18.907 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:23:18.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:23:18.910 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:23:18.910 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:23:18.910 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:23:18.910 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:23:18.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:18.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:18.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:18.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:18.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:19.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:19.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:19.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:19.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:19.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:19.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:19.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:19.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-10 zones={test-zone2, test-zone1} 2024-05-26T20:23:19.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:19.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:19.352 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:23:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:23:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:23:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:19.455+0000 7f0e65371a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:24.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:24.456+0000 7f0e65371a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:29.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:29.457+0000 7f0e65371a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:34.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:34.457+0000 7f0e65371a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:39.458+0000 7f0e65371a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:44.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:44.460+0000 7f0e65371a80 1 bucket sync caught up with source: 2024-05-26T20:23:44.460 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.5.6, , 00000000001.4.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2024-05-26T20:23:44.460 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.5.6, , 00000000001.4.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2024-05-26T20:23:44.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:44.460+0000 7f0e65371a80 0 bucket checkpoint complete 2024-05-26T20:23:44.476 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-10 zones={test-zone3, test-zone1} 2024-05-26T20:23:44.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:23:44.485 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:23:44.485 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:23:44.485 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:23:44.485 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:23:44.485 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:23:44.488 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:23:44.489 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:23:44.489 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:23:44.489 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:23:44.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:44.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:44.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:23:44.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:44.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:44.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:23:44.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:44.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:44.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:23:44.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:44.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:44.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:23:44.979 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-10 zones={test-zone3, test-zone1} 2024-05-26T20:23:44.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:44.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:23:44.980 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:23:45.044 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:23:45.044 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:23:45.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:45.083+0000 7f0cb0b53a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:50.084+0000 7f0cb0b53a80 1 waiting to reach incremental sync.. 2024-05-26T20:23:55.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:23:55.086+0000 7f0cb0b53a80 1 waiting to reach incremental sync.. 2024-05-26T20:24:00.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:00.087+0000 7f0cb0b53a80 1 bucket sync caught up with source: 2024-05-26T20:24:00.087 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.12.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.13.6, , ] 2024-05-26T20:24:00.087 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.12.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.13.6, , ] 2024-05-26T20:24:00.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:00.087+0000 7f0cb0b53a80 0 bucket checkpoint complete 2024-05-26T20:24:00.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-11 zones={test-zone1, test-zone2} 2024-05-26T20:24:00.115 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:00.122 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:00.122 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:00.122 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:00.122 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:00.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:00.125 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:00.126 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:00.126 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:00.126 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:00.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:00.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:00.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:00.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:00.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:00.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:00.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:00.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:00.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:00.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:00.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:00.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:00.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-11 zones={test-zone1, test-zone2} 2024-05-26T20:24:00.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:00.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:00.810 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:00.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:00.915+0000 7facc16e9a80 1 bucket sync caught up with source: 2024-05-26T20:24:00.915 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.12.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.13.6, , ] 2024-05-26T20:24:00.915 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.12.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.13.6, , ] 2024-05-26T20:24:00.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:00.915+0000 7facc16e9a80 0 bucket checkpoint complete 2024-05-26T20:24:00.928 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-11 zones={test-zone3, test-zone2} 2024-05-26T20:24:00.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:00.937 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:00.937 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:00.937 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:00.937 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:00.937 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:00.940 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:00.940 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:00.940 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:00.941 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:00.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:00.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:01.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:01.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-11 zones={test-zone3, test-zone2} 2024-05-26T20:24:01.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:01.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:01.385 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:01.480+0000 7fc67a713a80 1 bucket sync caught up with source: 2024-05-26T20:24:01.480 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.11.6, , 00000000001.9.6, , , , 00000000001.11.6, 00000000001.9.6, , ] 2024-05-26T20:24:01.480 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.11.6, , 00000000001.9.6, , , , 00000000001.11.6, 00000000001.9.6, , ] 2024-05-26T20:24:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:01.480+0000 7fc67a713a80 0 bucket checkpoint complete 2024-05-26T20:24:01.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-12 zones={test-zone1, test-zone3} 2024-05-26T20:24:01.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:01.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:01.503 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:01.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:01.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:01.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:01.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:01.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:01.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:01.861 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-12 zones={test-zone1, test-zone3} 2024-05-26T20:24:01.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:01.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:01.862 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:01.925 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:01.926 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:01.963 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:01.963+0000 7f711f5f0a80 1 bucket sync caught up with source: 2024-05-26T20:24:01.963 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.11.6, , 00000000001.9.6, , , , 00000000001.11.6, 00000000001.9.6, , ] 2024-05-26T20:24:01.963 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000001.11.6, , 00000000001.9.6, , , , 00000000001.11.6, 00000000001.9.6, , ] 2024-05-26T20:24:01.963 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:01.963+0000 7f711f5f0a80 0 bucket checkpoint complete 2024-05-26T20:24:01.975 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-12 zones={test-zone2, test-zone3} 2024-05-26T20:24:01.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:01.982 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:01.982 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:01.982 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:01.982 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:01.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-05-26T20:24:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-05-26T20:24:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-05-26T20:24:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:01.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:02.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:02.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-05-26T20:24:02.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:02.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:02.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-05-26T20:24:02.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:02.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:02.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-05-26T20:24:02.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:02.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:02.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:02.344 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-12 zones={test-zone2, test-zone3} 2024-05-26T20:24:02.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-05-26T20:24:02.346 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-13 2024-05-26T20:24:02.358 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-14 2024-05-26T20:24:02.380 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-15 2024-05-26T20:24:02.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.503 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:24:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:24:02.600 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:24:02.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755042.393398_73.1", 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:24:02.393398Z" 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.602 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:24:02.603 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:24:02.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.604 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:24:02.652 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:02.652 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:02.680 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:02.681+0000 7fbe638cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:24:02.681 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:02.682+0000 7fbe638cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:24:02.692 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:24:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:24:02.694 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:24:02.695 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755042.393398_73.1", 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:24:02.393398Z", 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:24:02.696 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:24:02.698 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:24:02.698 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:24:02.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:24:02.698 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:24:02Z" 2024-05-26T20:24:02.698 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:24:02.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:24:02.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:24:02.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:24:02.700 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:24:02.750 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:02.750 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:02.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:02.778+0000 7f5543974a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:24:02.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:02.779+0000 7f5543974a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:24:02.789 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:24:02.789 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:24:02.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:24:02.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755042.393398_73.1", 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:24:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:24:02.393398Z", 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:24:02.793 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:24:02Z" 2024-05-26T20:24:02.794 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:24:02.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:24:02.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:02.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:02.796 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:02.851 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:02.851 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:02.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:02.885+0000 7ff5b59c2a80 1 waiting to reach incremental sync.. 2024-05-26T20:24:07.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:07.886+0000 7ff5b59c2a80 1 waiting to reach incremental sync.. 2024-05-26T20:24:12.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:12.886+0000 7ff5b59c2a80 1 waiting to reach incremental sync.. 2024-05-26T20:24:17.887 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:17.887+0000 7ff5b59c2a80 1 bucket sync caught up with source: 2024-05-26T20:24:17.888 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:17.888 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:17.888 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:17.887+0000 7ff5b59c2a80 0 bucket checkpoint complete 2024-05-26T20:24:17.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-13 zones={test-zone2, test-zone1} 2024-05-26T20:24:17.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:17.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:17.955 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:17.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:17.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:18.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:18.082 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-13 zones={test-zone2, test-zone1} 2024-05-26T20:24:18.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:18.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:18.083 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:18.152 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:18.153 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:18.189+0000 7f8fd6835a80 1 waiting to reach incremental sync.. 2024-05-26T20:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.191+0000 7f8fd6835a80 1 bucket sync caught up with source: 2024-05-26T20:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.191+0000 7f8fd6835a80 0 bucket checkpoint complete 2024-05-26T20:24:23.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-13 zones={test-zone3, test-zone1} 2024-05-26T20:24:23.213 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:23.216 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:23.219 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-13 zones={test-zone3, test-zone1} 2024-05-26T20:24:23.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.294 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:23.356 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:23.356 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:23.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.394+0000 7fbd39e7da80 1 bucket sync caught up with source: 2024-05-26T20:24:23.394 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:24:23.394 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:24:23.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.394+0000 7fbd39e7da80 0 bucket checkpoint complete 2024-05-26T20:24:23.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-14 zones={test-zone1, test-zone2} 2024-05-26T20:24:23.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:23.413 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:23.416 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-14 zones={test-zone1, test-zone2} 2024-05-26T20:24:23.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.536 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.629+0000 7f6ffeeeea80 1 bucket sync caught up with source: 2024-05-26T20:24:23.629 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:24:23.629 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:24:23.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.629+0000 7f6ffeeeea80 0 bucket checkpoint complete 2024-05-26T20:24:23.641 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-14 zones={test-zone3, test-zone2} 2024-05-26T20:24:23.645 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:23.691 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:23.694 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-14 zones={test-zone3, test-zone2} 2024-05-26T20:24:23.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:23.816 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.909+0000 7f4666408a80 1 bucket sync caught up with source: 2024-05-26T20:24:23.909 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:24:23.909 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:24:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:23.909+0000 7f4666408a80 0 bucket checkpoint complete 2024-05-26T20:24:23.921 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-15 zones={test-zone1, test-zone3} 2024-05-26T20:24:23.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:23.927 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:23.930 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:23.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:23.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:24.041 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-15 zones={test-zone1, test-zone3} 2024-05-26T20:24:24.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:24.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:24.041 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:24.105 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:24.106 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:24.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:24.142+0000 7f63c0070a80 1 bucket sync caught up with source: 2024-05-26T20:24:24.143 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:24:24.143 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:24:24.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:24.142+0000 7f63c0070a80 0 bucket checkpoint complete 2024-05-26T20:24:24.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-15 zones={test-zone2, test-zone3} 2024-05-26T20:24:24.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:24.162 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:24.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:24.165 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-05-26T20:24:24.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:24.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:24.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-05-26T20:24:24.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-15 zones={test-zone2, test-zone3} 2024-05-26T20:24:24.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:24.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:24.291 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:24.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:24.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:24.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:24.385+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:24.386 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:24.386 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:29.387 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:29.386+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:29.387 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:29.387 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:34.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:34.388+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:34.388 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:34.388 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:39.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:39.389+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:39.389 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:39.389 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:44.390+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:44.391 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:44.391 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:49.391 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:49.391+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:49.392 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:49.392 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:54.392+0000 7fcf4b887a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:54.393 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-05-26T20:24:54.393 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:59.394 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:59.395+0000 7fcf4b887a80 1 bucket sync caught up with source: 2024-05-26T20:24:59.394 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:59.395 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:59.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:24:59.395+0000 7fcf4b887a80 0 bucket checkpoint complete 2024-05-26T20:24:59.409 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-13 zones={test-zone2, test-zone1} 2024-05-26T20:24:59.415 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:59.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:59.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-13 zones={test-zone2, test-zone1} 2024-05-26T20:24:59.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:59.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:59.426 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:59.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:59.528+0000 7f5ca0552a80 1 bucket sync caught up with source: 2024-05-26T20:24:59.528 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:59.529 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-05-26T20:24:59.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:59.528+0000 7f5ca0552a80 0 bucket checkpoint complete 2024-05-26T20:24:59.541 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-13 zones={test-zone3, test-zone1} 2024-05-26T20:24:59.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:24:59.547 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:24:59.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-13 zones={test-zone3, test-zone1} 2024-05-26T20:24:59.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:59.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:24:59.564 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:24:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:24:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:24:59.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:24:59.657+0000 7fa6934c8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:24:59.657 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:24:59.657 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:04.658+0000 7fa6934c8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:04.659 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:25:04.659 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:09.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:09.660+0000 7fa6934c8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:09.660 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:25:09.660 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:14.661+0000 7fa6934c8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:14.662 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-05-26T20:25:14.662 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:19.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:19.662+0000 7fa6934c8a80 1 bucket sync caught up with source: 2024-05-26T20:25:19.663 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:19.663 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:19.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:19.662+0000 7fa6934c8a80 0 bucket checkpoint complete 2024-05-26T20:25:19.679 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-14 zones={test-zone1, test-zone2} 2024-05-26T20:25:19.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:25:19.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:25:19.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-14 zones={test-zone1, test-zone2} 2024-05-26T20:25:19.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:19.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:19.738 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:25:19.803 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:19.803 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:19.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:19.841+0000 7f5b6444aa80 1 bucket sync caught up with source: 2024-05-26T20:25:19.841 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:19.842 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-05-26T20:25:19.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:19.841+0000 7f5b6444aa80 0 bucket checkpoint complete 2024-05-26T20:25:19.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-14 zones={test-zone3, test-zone2} 2024-05-26T20:25:19.859 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:25:19.862 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:25:19.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-14 zones={test-zone3, test-zone2} 2024-05-26T20:25:19.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:19.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:19.878 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:25:19.935 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:19.935 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:19.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:19.970+0000 7fa3e63d8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:19.970 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:25:19.970 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:24.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:24.971+0000 7fa3e63d8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:24.975 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:25:24.975 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:29.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:29.972+0000 7fa3e63d8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:29.973 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:25:29.973 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:34.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:34.974+0000 7fa3e63d8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:25:34.974 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-05-26T20:25:34.974 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:39.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:39.975+0000 7fa3e63d8a80 1 bucket sync caught up with source: 2024-05-26T20:25:39.975 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:39.975 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:39.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:39.975+0000 7fa3e63d8a80 0 bucket checkpoint complete 2024-05-26T20:25:39.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-15 zones={test-zone1, test-zone3} 2024-05-26T20:25:39.996 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:25:39.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:25:40.006 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-15 zones={test-zone1, test-zone3} 2024-05-26T20:25:40.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:40.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:40.007 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:25:40.071 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:40.072 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:41.112 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:41.112+0000 7f93bed34a80 1 bucket sync caught up with source: 2024-05-26T20:25:41.113 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:41.113 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2024-05-26T20:25:41.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:41.112+0000 7f93bed34a80 0 bucket checkpoint complete 2024-05-26T20:25:41.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-15 zones={test-zone2, test-zone3} 2024-05-26T20:25:41.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:25:41.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:25:41.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-15 zones={test-zone2, test-zone3} 2024-05-26T20:25:41.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-05-26T20:25:41.191 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-16 2024-05-26T20:25:41.207 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-17 2024-05-26T20:25:41.232 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-18 2024-05-26T20:25:44.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.489 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:25:44.542 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:44.543 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:44.574 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:25:44.586 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:25:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755141.249328_96.1", 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:25:41.249328Z" 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:25:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:25:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.588 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:25:44.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:25:44.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.589 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:25:44.646 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:44.646 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:44.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:44.676+0000 7fa4676aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:25:44.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:44.678+0000 7fa4676aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:25:44.689 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:25:44.689 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:25:44.690 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:25:44.691 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755141.249328_96.1", 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:25:44.692 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:25:41.249328Z", 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:25:44.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:25:44Z" 2024-05-26T20:25:44.694 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:25:44.695 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:25:44.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:25:44.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:25:44.696 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:25:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:44.777+0000 7fbf821b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:25:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:25:44.778+0000 7fbf821b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:25:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:25:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755141.249328_96.1", 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:25:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:25:41.249328Z", 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:25:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:25:44Z" 2024-05-26T20:25:44.794 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:25:44.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:25:44.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:44.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:25:44.796 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:25:44.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:25:44.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:25:44.888 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:44.887+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:25:49.888 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:49.888+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:25:54.889 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:54.889+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:25:59.889 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:25:59.890+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:04.891+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:09.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:09.891+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:14.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:14.891+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:19.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:19.892+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:24.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:24.892+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:29.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:29.893+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:34.894+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:39.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:39.894+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:44.895 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:44.895+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:49.896 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:49.895+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:54.896 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:54.897+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:26:59.897 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:26:59.898+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:04.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:04.899+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:09.898+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:14.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:14.899+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:19.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:19.899+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:24.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:24.900+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:29.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:29.900+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:34.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:34.901+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:39.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:39.901+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:44.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:44.902+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:49.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:49.904+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:54.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:54.904+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:27:59.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:27:59.905+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:04.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:04.906+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:09.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:09.905+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:14.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:14.906+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:19.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:19.906+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:24.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:24.907+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:29.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:29.907+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:34.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:34.908+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:39.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:39.909+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:44.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:44.909+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:49.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:49.911+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:54.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:54.911+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:28:59.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:28:59.912+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:04.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:04.912+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:09.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:09.912+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:14.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:14.913+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:19.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:19.913+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:24.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:24.914+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:29.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:29.915+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:34.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:34.915+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:39.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:39.916+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:44.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:44.917+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:49.918+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:54.918 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:54.919+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:29:59.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:29:59.918+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:04.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:04.919+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:09.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:09.920+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:14.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:14.920+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:19.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:19.921+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:24.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:24.921+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:29.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:29.922+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:34.922+0000 7fbcc40c8a80 1 waiting to reach incremental sync.. 2024-05-26T20:30:39.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:39.924+0000 7fbcc40c8a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T20:30:39.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:39.924+0000 7fbcc40c8a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T20:30:39.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:39.924+0000 7fbcc40c8a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T20:30:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T20:30:39.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-05-26T20:30:39.960 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-19 2024-05-26T20:30:39.979 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-20 2024-05-26T20:30:40.009 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-21 2024-05-26T20:30:40.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:40.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:40.061 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:40.114 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:40.114 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:40.146 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:30:40.171 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:30:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:30:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755440.059617_242.1", 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:30:40.059617Z" 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:40.172 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:30:40.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:30:40.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:40.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:40.174 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:40.232 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:40.232 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:40.264 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:40.265+0000 7fb82d408a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:40.264 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:40.266+0000 7fb82d408a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:30:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755439.970768_225.1", 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:30:39.970768Z", 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:30:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:30:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:30:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:30:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:30:40Z" 2024-05-26T20:30:40.282 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:30:40.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716755440.059617_242.1 target=1_1716755439.970768_225.1 2024-05-26T20:30:45.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:45.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:45.284 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:45.341 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:45.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:45.374 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:45.375+0000 7f0a49ccfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:45.375 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:45.376+0000 7f0a49ccfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:45.387 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:30:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:30:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:30:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:30:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755440.027419_235.1", 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:30:40.027419Z", 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:30:45Z" 2024-05-26T20:30:45.392 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:30:45.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716755440.059617_242.1 target=1_1716755440.027419_235.1 2024-05-26T20:30:50.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:50.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:50.394 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:50.453 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:50.453 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:50.486 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:50.486+0000 7ff49ba32a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:50.486 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:50.488+0000 7ff49ba32a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:50.498 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:50.499 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:50.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755440.059617_242.1", 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:50.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:30:40.059617Z", 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:30:50.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:30:50Z" 2024-05-26T20:30:50.503 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:30:50.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:30:50.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:30:50.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:50.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:50.505 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:50.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:30:50.594+0000 7fb3f2b2ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:30:50.595+0000 7fb3f2b2ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:50.607 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:30:50.607 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:30:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755439.991476_229.1", 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:30:39.991476Z", 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:30:50Z" 2024-05-26T20:30:50.613 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:30:50.614 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716755440.059617_242.1 target=1_1716755439.991476_229.1 2024-05-26T20:30:55.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:55.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:30:55.616 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:30:55.680 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:55.680 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:30:55.711+0000 7f2d6e51ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:30:55.712+0000 7f2d6e51ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:30:55.723 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:30:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:30:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:30:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:30:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:55.725 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755440.059617_242.1", 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:30:40.059617Z", 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:55.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:30:55.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:30:55Z" 2024-05-26T20:30:55.728 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:30:55.729 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:30:55.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:30:55.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:30:55.741 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:30:55.805 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:30:55.805 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:30:55.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:30:55.844+0000 7fb37987fa80 1 waiting to reach incremental sync.. 2024-05-26T20:31:00.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:00.846+0000 7fb37987fa80 1 bucket sync caught up with source: 2024-05-26T20:31:00.847 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-05-26T20:31:00.847 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2024-05-26T20:31:00.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:00.846+0000 7fb37987fa80 0 bucket checkpoint complete 2024-05-26T20:31:00.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:00.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:00.863 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:00.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:00.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:00.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:00.966+0000 7f97465b4a80 1 waiting to reach incremental sync.. 2024-05-26T20:31:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:05.968+0000 7f97465b4a80 1 bucket sync caught up with source: 2024-05-26T20:31:05.969 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-05-26T20:31:05.969 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2024-05-26T20:31:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:05.968+0000 7f97465b4a80 0 bucket checkpoint complete 2024-05-26T20:31:06.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:06.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:06.001 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:06.066 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:06.066 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:06.104+0000 7ff4a6d6da80 1 waiting to reach incremental sync.. 2024-05-26T20:31:11.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:11.104+0000 7ff4a6d6da80 1 waiting to reach incremental sync.. 2024-05-26T20:31:16.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:16.105+0000 7ff4a6d6da80 1 waiting to reach incremental sync.. 2024-05-26T20:31:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:21.107+0000 7ff4a6d6da80 1 bucket sync caught up with source: 2024-05-26T20:31:21.107 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-05-26T20:31:21.107 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000002.48.11, , , , , ] 2024-05-26T20:31:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:21.107+0000 7ff4a6d6da80 0 bucket checkpoint complete 2024-05-26T20:31:21.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:21.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:21.123 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:21.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:21.223+0000 7f83b2356a80 1 waiting to reach incremental sync.. 2024-05-26T20:31:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:26.225+0000 7f83b2356a80 1 bucket sync caught up with source: 2024-05-26T20:31:26.225 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.50.10, , , , , ] 2024-05-26T20:31:26.225 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.50.10, , , , , ] 2024-05-26T20:31:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:26.225+0000 7f83b2356a80 0 bucket checkpoint complete 2024-05-26T20:31:26.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:26.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:26.256 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:26.319 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:26.353+0000 7fdfb5764a80 1 waiting to reach incremental sync.. 2024-05-26T20:31:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:31.354+0000 7fdfb5764a80 1 waiting to reach incremental sync.. 2024-05-26T20:31:36.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:36.355+0000 7fdfb5764a80 1 waiting to reach incremental sync.. 2024-05-26T20:31:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:41.356+0000 7fdfb5764a80 1 bucket sync caught up with source: 2024-05-26T20:31:41.357 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.108.11, , , , , ] 2024-05-26T20:31:41.357 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000002.108.11, , , , , ] 2024-05-26T20:31:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:31:41.356+0000 7fdfb5764a80 0 bucket checkpoint complete 2024-05-26T20:31:41.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:41.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:41.374 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:41.437 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:41.437 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:41.476 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:41.477+0000 7f9b109dca80 1 bucket sync caught up with source: 2024-05-26T20:31:41.476 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000002.108.11, , , , , ] 2024-05-26T20:31:41.476 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000002.108.11, , , , , ] 2024-05-26T20:31:41.476 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:41.477+0000 7f9b109dca80 0 bucket checkpoint complete 2024-05-26T20:31:42.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:42.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:31:42.271 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:31:42.327 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:31:42.327 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:31:42.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:42.364+0000 7f1c82713a80 1 waiting for incremental sync to catch up: 2024-05-26T20:31:42.363 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:42.363 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.29.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:47.364 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:47.365+0000 7f1c82713a80 1 waiting for incremental sync to catch up: 2024-05-26T20:31:47.365 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:47.365 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.29.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:52.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:52.365+0000 7f1c82713a80 1 waiting for incremental sync to catch up: 2024-05-26T20:31:52.366 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:52.366 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.29.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:57.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:31:57.366+0000 7f1c82713a80 1 waiting for incremental sync to catch up: 2024-05-26T20:31:57.370 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.33.10, , , , , ] 2024-05-26T20:31:57.370 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.29.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:32:02.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:02.367+0000 7f1c82713a80 1 bucket sync caught up with source: 2024-05-26T20:32:02.367 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000005.33.10, , , , , ] 2024-05-26T20:32:02.367 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.29.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:32:02.367 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:02.367+0000 7f1c82713a80 0 bucket checkpoint complete 2024-05-26T20:32:02.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:02.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:02.384 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:02.449 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:02.450 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:02.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:02.488+0000 7f3efdbf4a80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:02.488 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.12.6, , , , 00000000005.33.10, , , , , ] 2024-05-26T20:32:02.488 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:07.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:07.489+0000 7f3efdbf4a80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:07.490 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000005.33.10, , , , , ] 2024-05-26T20:32:07.490 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:12.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:12.491+0000 7f3efdbf4a80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:12.491 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000005.33.10, , , , , ] 2024-05-26T20:32:12.491 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:17.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:17.492+0000 7f3efdbf4a80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:17.493 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000005.33.10, , , , , ] 2024-05-26T20:32:17.493 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:22.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:22.493+0000 7f3efdbf4a80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:22.493 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000005.33.10, , , , , ] 2024-05-26T20:32:22.493 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:27.495+0000 7f3efdbf4a80 1 bucket sync caught up with source: 2024-05-26T20:32:27.495 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:27.496 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.29.6, , 00000000003.54.10, , 00000000007.37.11, , , , , ] 2024-05-26T20:32:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:27.495+0000 7f3efdbf4a80 0 bucket checkpoint complete 2024-05-26T20:32:27.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:27.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:27.511 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:27.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:27.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:27.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:27.614+0000 7f52f1a1da80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:27.614 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000010.89.6, , 00000000004.68.10, , , , , ] 2024-05-26T20:32:27.614 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:32.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:32.616+0000 7f52f1a1da80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:32.616 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000010.89.6, , 00000000004.68.10, , , , , ] 2024-05-26T20:32:32.616 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:37.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:37.617+0000 7f52f1a1da80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:37.617 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000010.89.6, , 00000000004.68.10, , , , , ] 2024-05-26T20:32:37.617 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:42.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.618+0000 7f52f1a1da80 1 bucket sync caught up with source: 2024-05-26T20:32:42.619 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:42.619 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.618+0000 7f52f1a1da80 0 bucket checkpoint complete 2024-05-26T20:32:42.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.635 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:42.699 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:42.699 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:42.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.738+0000 7f0ef085fa80 1 bucket sync caught up with source: 2024-05-26T20:32:42.738 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:42.738 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000010.89.6, , 00000000006.72.11, , , , , ] 2024-05-26T20:32:42.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.738+0000 7f0ef085fa80 0 bucket checkpoint complete 2024-05-26T20:32:42.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.751 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:42.809 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:42.809 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.843+0000 7fae82db5a80 1 bucket sync caught up with source: 2024-05-26T20:32:42.843 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.199.10, , 00000000003.106.10, , 00000000014.146.6, , , , , ] 2024-05-26T20:32:42.843 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.199.10, , 00000000003.106.10, , 00000000014.146.6, , , , , ] 2024-05-26T20:32:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:42.843+0000 7fae82db5a80 0 bucket checkpoint complete 2024-05-26T20:32:42.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:42.855 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:42.926 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:42.926 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:42.975 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:42.976+0000 7f33e6ff5a80 1 bucket sync caught up with source: 2024-05-26T20:32:42.975 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.199.10, , 00000000003.106.10, , 00000000014.146.6, , , , , ] 2024-05-26T20:32:42.975 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.199.10, , 00000000003.106.10, , 00000000014.146.6, , , , , ] 2024-05-26T20:32:42.976 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:42.976+0000 7f33e6ff5a80 0 bucket checkpoint complete 2024-05-26T20:32:42.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone2, test-zone1} 2024-05-26T20:32:42.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:43.001 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:43.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:43.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:43.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:43.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:43.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:43.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:43.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:43.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:43.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:43.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:43.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:43.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:43.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:43.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:43.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone2, test-zone1} 2024-05-26T20:32:43.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone3, test-zone1} 2024-05-26T20:32:43.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:43.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:43.646 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:43.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:43.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:43.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:43.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:43.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:43.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:43.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:43.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:44.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone3, test-zone1} 2024-05-26T20:32:44.191 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone3, test-zone2} 2024-05-26T20:32:44.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:44.198 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:44.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:44.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:44.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:44.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:44.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:44.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:44.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:44.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:44.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:44.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:44.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:44.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:44.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:44.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:44.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:44.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:44.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:44.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:44.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone3, test-zone2} 2024-05-26T20:32:44.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:44.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:44.835 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:44.897 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:44.897 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:44.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:44.936+0000 7f14b0a99a80 1 bucket sync caught up with source: 2024-05-26T20:32:44.935 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.24.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:32:44.936 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.24.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:32:44.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:44.936+0000 7f14b0a99a80 0 bucket checkpoint complete 2024-05-26T20:32:44.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:44.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:44.948 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.052+0000 7f44badada80 1 bucket sync caught up with source: 2024-05-26T20:32:45.052 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.24.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:32:45.052 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.24.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:32:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.052+0000 7f44badada80 0 bucket checkpoint complete 2024-05-26T20:32:45.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.065 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:45.125 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:45.125 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:45.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.161+0000 7f10c016ca80 1 bucket sync caught up with source: 2024-05-26T20:32:45.161 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.36.10, , 00000000010.53.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:32:45.161 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.36.10, , 00000000010.53.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:32:45.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.161+0000 7f10c016ca80 0 bucket checkpoint complete 2024-05-26T20:32:45.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.173 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:45.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:45.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:45.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.265+0000 7f2fe614aa80 1 bucket sync caught up with source: 2024-05-26T20:32:45.265 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.36.10, , 00000000010.53.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:32:45.265 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.36.10, , 00000000010.53.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:32:45.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.265+0000 7f2fe614aa80 0 bucket checkpoint complete 2024-05-26T20:32:45.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.277 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:45.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:45.335 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:45.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.371+0000 7f8cedc71a80 1 bucket sync caught up with source: 2024-05-26T20:32:45.371 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.65.10, , 00000000003.70.10, , 00000000014.105.6, , , , , ] 2024-05-26T20:32:45.371 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.65.10, , 00000000003.70.10, , 00000000014.105.6, , , , , ] 2024-05-26T20:32:45.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:32:45.371+0000 7f8cedc71a80 0 bucket checkpoint complete 2024-05-26T20:32:45.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:45.384 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:45.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:45.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:45.475 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:45.476+0000 7fcbf13cba80 1 bucket sync caught up with source: 2024-05-26T20:32:45.475 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.65.10, , 00000000003.70.10, , 00000000014.105.6, , , , , ] 2024-05-26T20:32:45.476 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.65.10, , 00000000003.70.10, , 00000000014.105.6, , , , , ] 2024-05-26T20:32:45.476 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:45.476+0000 7fcbf13cba80 0 bucket checkpoint complete 2024-05-26T20:32:45.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone2, test-zone1} 2024-05-26T20:32:45.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:45.495 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:45.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:45.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:45.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:45.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:45.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:45.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:45.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:45.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:45.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:45.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:45.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:45.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:45.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:45.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:45.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:46.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:46.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone2, test-zone1} 2024-05-26T20:32:46.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone3, test-zone1} 2024-05-26T20:32:46.291 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:46.294 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:46.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:46.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:46.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:46.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:46.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:46.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:46.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:46.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:46.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:46.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:46.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:46.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:46.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:46.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:46.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:46.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:47.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone3, test-zone1} 2024-05-26T20:32:47.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone3, test-zone2} 2024-05-26T20:32:47.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:32:47.026 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:47.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:47.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:47.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:47.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:32:47.029 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:32:47.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:32:47.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:32:47.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:32:47.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:47.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:47.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:32:47.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:47.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:47.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:32:47.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:47.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:47.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:32:47.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:47.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:47.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:32:47.755 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone3, test-zone2} 2024-05-26T20:32:47.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:47.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:32:47.755 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:32:47.820 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:32:47.820 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:32:47.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:47.858+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:47.857 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:32:47.857 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:32:52.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:52.858+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:52.859 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:32:52.859 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:32:57.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:32:57.860+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:32:57.860 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:32:57.860 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:02.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:02.861+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:02.862 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:02.862 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:07.862+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:07.863 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:07.863 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:12.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:12.864+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:12.864 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:12.864 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:17.865 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:17.865+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:17.865 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:17.865 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:22.866 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:22.866+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:22.869 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:22.869 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:27.868+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:27.867 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:27.868 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:32.869 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:32.869+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:32.869 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:32.869 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:37.870 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:37.871+0000 7f5cb53bda80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:37.870 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000004.52.12, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:37.870 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:42.871 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:42.872+0000 7f5cb53bda80 1 bucket sync caught up with source: 2024-05-26T20:33:42.871 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:42.871 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:42.871 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:42.872+0000 7f5cb53bda80 0 bucket checkpoint complete 2024-05-26T20:33:42.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:42.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:42.888 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:42.954 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:42.954 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:42.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:42.992+0000 7fc89dc60a80 1 bucket sync caught up with source: 2024-05-26T20:33:42.992 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:42.992 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000010.64.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:33:42.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:42.992+0000 7fc89dc60a80 0 bucket checkpoint complete 2024-05-26T20:33:43.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.006 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.099+0000 7ffb9806aa80 1 bucket sync caught up with source: 2024-05-26T20:33:43.099 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.85.10, , 00000000010.67.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:33:43.099 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.85.10, , 00000000010.67.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:33:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.099+0000 7ffb9806aa80 0 bucket checkpoint complete 2024-05-26T20:33:43.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.112 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:43.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.204+0000 7fa2ab029a80 1 bucket sync caught up with source: 2024-05-26T20:33:43.205 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.85.10, , 00000000010.67.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:33:43.205 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.85.10, , 00000000010.67.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:33:43.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.204+0000 7fa2ab029a80 0 bucket checkpoint complete 2024-05-26T20:33:43.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.217 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:43.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.309+0000 7fe5a4872a80 1 bucket sync caught up with source: 2024-05-26T20:33:43.310 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.107.10, , 00000000003.103.10, , 00000000014.132.10, , , , , ] 2024-05-26T20:33:43.310 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.107.10, , 00000000003.103.10, , 00000000014.132.10, , , , , ] 2024-05-26T20:33:43.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:33:43.309+0000 7fe5a4872a80 0 bucket checkpoint complete 2024-05-26T20:33:43.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:43.322 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:43.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:43.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:43.425 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:43.426+0000 7f57308a0a80 1 bucket sync caught up with source: 2024-05-26T20:33:43.425 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.107.10, , 00000000003.103.10, , 00000000014.132.10, , , , , ] 2024-05-26T20:33:43.425 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.107.10, , 00000000003.103.10, , 00000000014.132.10, , , , , ] 2024-05-26T20:33:43.426 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:43.426+0000 7f57308a0a80 0 bucket checkpoint complete 2024-05-26T20:33:43.438 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone2, test-zone1} 2024-05-26T20:33:43.444 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:33:43.447 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:43.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:43.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:43.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:43.448 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:33:43.451 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:43.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:43.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:43.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:43.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:43.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:43.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:43.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:43.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:43.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:43.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:43.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:43.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:43.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:43.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:44.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:44.224 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone2, test-zone1} 2024-05-26T20:33:44.224 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone3, test-zone1} 2024-05-26T20:33:44.228 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:33:44.231 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:44.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:44.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:44.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:44.232 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:33:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:44.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:44.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:44.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:44.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:44.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:44.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:44.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:44.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:44.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:44.779 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone3, test-zone1} 2024-05-26T20:33:44.780 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone3, test-zone2} 2024-05-26T20:33:44.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:33:44.787 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:44.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:44.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:44.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:44.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:33:44.790 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:33:44.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:33:44.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:33:44.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:33:44.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:33:44.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:44.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:45.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:33:45.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:45.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:45.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:33:45.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:45.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:45.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:33:45.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone3, test-zone2} 2024-05-26T20:33:45.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:45.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:33:45.698 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:33:45.761 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:33:45.761 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:33:45.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:45.798+0000 7f8343448a80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:45.798 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:33:45.798 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:33:50.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:50.799+0000 7f8343448a80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:50.799 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:33:50.800 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:33:55.801 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:33:55.800+0000 7f8343448a80 1 waiting for incremental sync to catch up: 2024-05-26T20:33:55.801 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:33:55.801 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:00.802 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:00.802+0000 7f8343448a80 1 bucket sync caught up with source: 2024-05-26T20:34:00.803 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:00.803 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:00.803 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:00.802+0000 7f8343448a80 0 bucket checkpoint complete 2024-05-26T20:34:00.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:00.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:00.818 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:00.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:00.922+0000 7f4ef60a3a80 1 waiting for incremental sync to catch up: 2024-05-26T20:34:00.922 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:00.922 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:05.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:05.924+0000 7f4ef60a3a80 1 waiting for incremental sync to catch up: 2024-05-26T20:34:05.924 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000010.29.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:05.924 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:10.925+0000 7f4ef60a3a80 1 bucket sync caught up with source: 2024-05-26T20:34:10.925 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:10.926 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000011.32.6, , 00000000003.54.10, , 00000000008.39.10, , , , , ] 2024-05-26T20:34:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:10.925+0000 7f4ef60a3a80 0 bucket checkpoint complete 2024-05-26T20:34:10.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:10.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:10.941 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:11.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.042+0000 7f34013a9a80 1 bucket sync caught up with source: 2024-05-26T20:34:11.042 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000011.92.6, , 00000000007.74.10, , , , , ] 2024-05-26T20:34:11.042 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000011.92.6, , 00000000007.74.10, , , , , ] 2024-05-26T20:34:11.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.042+0000 7f34013a9a80 0 bucket checkpoint complete 2024-05-26T20:34:11.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.054 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:11.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.150+0000 7fe327a66a80 1 bucket sync caught up with source: 2024-05-26T20:34:11.151 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.81.10, , 00000000011.92.6, , 00000000007.74.10, , , , , ] 2024-05-26T20:34:11.151 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.81.10, , 00000000011.92.6, , 00000000007.74.10, , , , , ] 2024-05-26T20:34:11.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.150+0000 7fe327a66a80 0 bucket checkpoint complete 2024-05-26T20:34:11.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.163 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.257+0000 7f6fc6dcfa80 1 bucket sync caught up with source: 2024-05-26T20:34:11.257 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.199.10, , 00000000003.106.10, , 00000000015.149.6, , , , , ] 2024-05-26T20:34:11.257 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.199.10, , 00000000003.106.10, , 00000000015.149.6, , , , , ] 2024-05-26T20:34:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:11.257+0000 7f6fc6dcfa80 0 bucket checkpoint complete 2024-05-26T20:34:11.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:11.270 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:11.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:11.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:11.373 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:11.373+0000 7f1d560c8a80 1 bucket sync caught up with source: 2024-05-26T20:34:11.374 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.199.10, , 00000000003.106.10, , 00000000015.149.6, , , , , ] 2024-05-26T20:34:11.374 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.199.10, , 00000000003.106.10, , 00000000015.149.6, , , , , ] 2024-05-26T20:34:11.374 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:11.373+0000 7f1d560c8a80 0 bucket checkpoint complete 2024-05-26T20:34:11.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone2, test-zone1} 2024-05-26T20:34:11.391 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:11.394 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:11.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:11.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:11.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:11.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:11.397 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:11.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:11.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:11.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:11.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:11.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:11.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:11.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:11.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:11.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:11.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:11.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:11.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:11.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:12.075 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone2, test-zone1} 2024-05-26T20:34:12.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone3, test-zone1} 2024-05-26T20:34:12.079 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:12.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:12.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:12.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:12.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:12.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:12.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:12.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:12.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:12.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:12.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:12.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone3, test-zone1} 2024-05-26T20:34:12.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-19 zones={test-zone3, test-zone2} 2024-05-26T20:34:12.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:12.770 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:12.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:12.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:12.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:12.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:12.773 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:12.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:12.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:13.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:13.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:13.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:13.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:13.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:13.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:13.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:13.635 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-19 zones={test-zone3, test-zone2} 2024-05-26T20:34:13.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.635 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:13.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:13.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:13.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:13.737+0000 7fa97f6aea80 1 bucket sync caught up with source: 2024-05-26T20:34:13.737 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000011.27.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:34:13.737 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.27.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:34:13.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:13.737+0000 7fa97f6aea80 0 bucket checkpoint complete 2024-05-26T20:34:13.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.751 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:13.817 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:13.817 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:13.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:13.857+0000 7f052099aa80 1 bucket sync caught up with source: 2024-05-26T20:34:13.857 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000011.27.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:34:13.857 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000011.27.6, , 00000000003.11.10, , 00000000007.35.10, , , , , ] 2024-05-26T20:34:13.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:13.857+0000 7f052099aa80 0 bucket checkpoint complete 2024-05-26T20:34:13.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.870 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:13.963+0000 7f26397baa80 1 bucket sync caught up with source: 2024-05-26T20:34:13.964 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.36.10, , 00000000011.56.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:34:13.964 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.36.10, , 00000000011.56.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:34:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:13.963+0000 7f26397baa80 0 bucket checkpoint complete 2024-05-26T20:34:13.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:13.976 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:14.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:14.068+0000 7f95045d9a80 1 bucket sync caught up with source: 2024-05-26T20:34:14.068 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.36.10, , 00000000011.56.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:34:14.068 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.36.10, , 00000000011.56.6, , 00000000008.60.10, , , , , ] 2024-05-26T20:34:14.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:14.068+0000 7f95045d9a80 0 bucket checkpoint complete 2024-05-26T20:34:14.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:14.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:14.080 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:14.137 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:14.138 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:14.174+0000 7feca3c84a80 1 bucket sync caught up with source: 2024-05-26T20:34:14.174 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.65.10, , 00000000003.70.10, , 00000000015.108.6, , , , , ] 2024-05-26T20:34:14.174 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.65.10, , 00000000003.70.10, , 00000000015.108.6, , , , , ] 2024-05-26T20:34:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:14.174+0000 7feca3c84a80 0 bucket checkpoint complete 2024-05-26T20:34:14.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:14.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:14.187 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:14.249 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:14.249 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:14.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:14.287+0000 7f845eddaa80 1 bucket sync caught up with source: 2024-05-26T20:34:14.287 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.65.10, , 00000000003.70.10, , 00000000015.108.6, , , , , ] 2024-05-26T20:34:14.287 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.65.10, , 00000000003.70.10, , 00000000015.108.6, , , , , ] 2024-05-26T20:34:14.288 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:14.287+0000 7f845eddaa80 0 bucket checkpoint complete 2024-05-26T20:34:14.300 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone2, test-zone1} 2024-05-26T20:34:14.305 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:14.308 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:14.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:14.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:14.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:14.308 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:14.311 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:14.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:14.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:14.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:14.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:14.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:14.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:14.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:14.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:14.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:14.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:14.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:14.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:14.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:14.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:14.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:15.080 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone2, test-zone1} 2024-05-26T20:34:15.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone3, test-zone1} 2024-05-26T20:34:15.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:15.087 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:15.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:15.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:15.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:15.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:15.090 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:15.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:15.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:15.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:15.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:15.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:15.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:15.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:15.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:15.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:15.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:15.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:15.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:15.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone3, test-zone1} 2024-05-26T20:34:15.727 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-20 zones={test-zone3, test-zone2} 2024-05-26T20:34:15.732 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:15.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:15.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:15.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:15.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:15.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:15.738 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:15.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:15.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:15.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:15.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:15.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:15.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:16.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:16.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:16.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:16.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:16.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:16.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:16.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:16.595 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-20 zones={test-zone3, test-zone2} 2024-05-26T20:34:16.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.595 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:16.659 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:16.660 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:16.698 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:16.698+0000 7f4638797a80 1 bucket sync caught up with source: 2024-05-26T20:34:16.699 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000011.67.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:34:16.699 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000011.67.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:34:16.699 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:16.698+0000 7f4638797a80 0 bucket checkpoint complete 2024-05-26T20:34:16.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.712 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:16.817+0000 7f1fc9075a80 1 bucket sync caught up with source: 2024-05-26T20:34:16.817 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000011.67.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:34:16.817 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000011.67.6, , 00000000003.11.10, , 00000000007.18.10, , , , , ] 2024-05-26T20:34:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:16.817+0000 7f1fc9075a80 0 bucket checkpoint complete 2024-05-26T20:34:16.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.831 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:16.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:16.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:16.925+0000 7f4628203a80 1 bucket sync caught up with source: 2024-05-26T20:34:16.925 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.85.10, , 00000000011.70.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:34:16.925 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.85.10, , 00000000011.70.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:34:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:16.925+0000 7f4628203a80 0 bucket checkpoint complete 2024-05-26T20:34:16.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:16.937 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:17.029+0000 7fa8fc5bca80 1 bucket sync caught up with source: 2024-05-26T20:34:17.029 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.85.10, , 00000000011.70.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:34:17.029 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.85.10, , 00000000011.70.6, , 00000000007.57.10, , , , , ] 2024-05-26T20:34:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:17.029+0000 7fa8fc5bca80 0 bucket checkpoint complete 2024-05-26T20:34:17.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:17.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:17.041 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:17.099 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:17.100 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:17.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:17.136+0000 7f8574adaa80 1 bucket sync caught up with source: 2024-05-26T20:34:17.136 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000003.107.10, , 00000000003.103.10, , 00000000015.135.6, , , , , ] 2024-05-26T20:34:17.136 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000003.107.10, , 00000000003.103.10, , 00000000015.135.6, , , , , ] 2024-05-26T20:34:17.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:17.136+0000 7f8574adaa80 0 bucket checkpoint complete 2024-05-26T20:34:17.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:17.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:17.148 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:17.213 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:17.213 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:17.250 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:17.250+0000 7fdad04fea80 1 bucket sync caught up with source: 2024-05-26T20:34:17.250 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, 00000000003.107.10, , 00000000003.103.10, , 00000000015.135.6, , , , , ] 2024-05-26T20:34:17.250 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, 00000000003.107.10, , 00000000003.103.10, , 00000000015.135.6, , , , , ] 2024-05-26T20:34:17.251 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:17.250+0000 7fdad04fea80 0 bucket checkpoint complete 2024-05-26T20:34:17.263 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone2, test-zone1} 2024-05-26T20:34:17.267 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:17.270 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:17.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:17.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:17.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:17.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:17.273 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:17.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:17.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:17.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:17.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:17.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:17.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:17.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:17.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:17.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:17.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:17.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:17.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:17.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:17.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:18.122 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone2, test-zone1} 2024-05-26T20:34:18.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone3, test-zone1} 2024-05-26T20:34:18.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:18.129 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:18.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:18.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:18.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:18.129 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:18.132 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:18.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:18.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:18.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:18.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:18.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:18.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:18.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:18.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:18.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:18.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:18.683 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone3, test-zone1} 2024-05-26T20:34:18.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-21 zones={test-zone3, test-zone2} 2024-05-26T20:34:18.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:34:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:18.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:34:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:34:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-05-26T20:34:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-05-26T20:34:18.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-05-26T20:34:18.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:34:18.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:18.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:19.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-05-26T20:34:19.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:19.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:19.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-05-26T20:34:19.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:19.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:19.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-05-26T20:34:19.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-21 zones={test-zone3, test-zone2} 2024-05-26T20:34:19.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-05-26T20:34:19.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.538 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:34:19.593 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:19.594 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755659.538545_364.1", 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:34:19.538545Z" 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.640 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:34:19.641 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:34:19.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.641 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:34:19.702 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:19.702 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:19.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:19.735+0000 7f978bc61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:19.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:19.736+0000 7f978bc61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.748 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755659.538545_364.1", 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:34:19.538545Z", 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:34:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:34:19.753 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:34:19.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:34:19.753 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:34:19Z" 2024-05-26T20:34:19.753 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:34:19.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:34:19.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:34:19.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:19.754 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:34:19.806 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:19.806 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:19.835+0000 7f7b0af07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:19.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:19.836+0000 7f7b0af07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.847 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:19.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755659.524663_359.1", 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:34:19.524663Z", 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:34:19.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:34:19Z" 2024-05-26T20:34:19.852 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:34:19.853 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716755659.538545_364.1 target=1_1716755659.524663_359.1 2024-05-26T20:34:24.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:24.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:34:24.857 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:34:24.916 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:24.916 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:24.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:24.948+0000 7f47cd554a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:24.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:24.949+0000 7f47cd554a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:34:24.961 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:34:24.961 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:34:24.961 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:34:24.961 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:34:24.961 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:34:24.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755659.538545_364.1", 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:34:19.538545Z", 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:24.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:24.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:34:24Z" 2024-05-26T20:34:24.966 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:34:24.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:34:24.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:24.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:24.980 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:25.044 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:25.044 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:25.081 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:25.082+0000 7f06cde2da80 1 waiting to reach incremental sync.. 2024-05-26T20:34:30.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:30.083+0000 7f06cde2da80 1 waiting to reach incremental sync.. 2024-05-26T20:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:35.084+0000 7f06cde2da80 1 waiting to reach incremental sync.. 2024-05-26T20:34:40.084 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:40.084+0000 7f06cde2da80 1 waiting to reach incremental sync.. 2024-05-26T20:34:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:45.085+0000 7f06cde2da80 1 bucket sync caught up with source: 2024-05-26T20:34:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000003.15.10, , , , , ] 2024-05-26T20:34:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000002.13.11, , , , , ] 2024-05-26T20:34:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:34:45.085+0000 7f06cde2da80 0 bucket checkpoint complete 2024-05-26T20:34:45.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:45.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:45.102 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:45.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:45.205+0000 7fe1574b8a80 1 waiting for incremental sync to catch up: 2024-05-26T20:34:45.205 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2024-05-26T20:34:45.205 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.15.10, , , , , ] 2024-05-26T20:34:50.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:50.206+0000 7fe1574b8a80 1 bucket sync caught up with source: 2024-05-26T20:34:50.211 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.15.10, , , , , ] 2024-05-26T20:34:50.211 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.15.10, , , , , ] 2024-05-26T20:34:50.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:50.206+0000 7fe1574b8a80 0 bucket checkpoint complete 2024-05-26T20:34:50.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:50.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:34:50.223 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:34:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:34:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:34:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:50.325+0000 7fac4fb5ea80 1 waiting to reach incremental sync.. 2024-05-26T20:34:55.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:34:55.325+0000 7fac4fb5ea80 1 waiting to reach incremental sync.. 2024-05-26T20:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:00.327+0000 7fac4fb5ea80 1 bucket sync caught up with source: 2024-05-26T20:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.57.11, , , , , ] 2024-05-26T20:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000002.57.11, , , , , ] 2024-05-26T20:35:00.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:00.327+0000 7fac4fb5ea80 0 bucket checkpoint complete 2024-05-26T20:35:00.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:00.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:00.341 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:35:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:35:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:35:00.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:00.439+0000 7f4552837a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:00.439 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.57.11, , , , , ] 2024-05-26T20:35:00.439 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , , , ] 2024-05-26T20:35:05.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:05.440+0000 7f4552837a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:05.440 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.57.11, , , , , ] 2024-05-26T20:35:05.441 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , , , ] 2024-05-26T20:35:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:10.442+0000 7f4552837a80 1 bucket sync caught up with source: 2024-05-26T20:35:10.442 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , , , ] 2024-05-26T20:35:10.442 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , , , ] 2024-05-26T20:35:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:10.442+0000 7f4552837a80 0 bucket checkpoint complete 2024-05-26T20:35:10.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:10.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:10.458 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:35:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:35:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:35:10.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:10.560+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:10.560 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:10.560 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:15.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:15.561+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:15.562 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:15.562 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:20.563+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:20.563 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:20.563 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:25.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:25.564+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:25.564 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:25.564 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:30.565+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:30.566 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:30.566 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:35.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:35.567+0000 7f1999a82a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:35.567 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000002.114.11, , , , , ] 2024-05-26T20:35:35.567 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:40.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:40.568+0000 7f1999a82a80 1 bucket sync caught up with source: 2024-05-26T20:35:40.568 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:40.568 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:40.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:35:40.568+0000 7f1999a82a80 0 bucket checkpoint complete 2024-05-26T20:35:40.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:40.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:40.586 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:35:40.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:35:40.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:35:40.689 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:40.689+0000 7f72658b9a80 1 bucket sync caught up with source: 2024-05-26T20:35:40.689 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:40.689 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000004.118.10, , , , , ] 2024-05-26T20:35:40.689 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:40.689+0000 7f72658b9a80 0 bucket checkpoint complete 2024-05-26T20:35:40.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone2, test-zone1} 2024-05-26T20:35:40.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:35:40.715 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:35:40.717 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone2, test-zone1} 2024-05-26T20:35:40.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone3, test-zone1} 2024-05-26T20:35:40.797 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:35:40.799 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:35:40.802 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone3, test-zone1} 2024-05-26T20:35:40.969 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone3, test-zone2} 2024-05-26T20:35:40.974 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:35:40.977 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.977 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:35:40.979 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:35:40.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:40.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:41.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:35:41.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone3, test-zone2} 2024-05-26T20:35:41.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:41.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:35:41.703 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:35:41.759 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:35:41.760 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:35:41.793 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:41.793+0000 7f50948a0a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:41.793 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.19.10, , , , , ] 2024-05-26T20:35:41.794 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:35:46.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:46.794+0000 7f50948a0a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:46.795 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.19.10, , , , , ] 2024-05-26T20:35:46.795 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:35:51.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:51.795+0000 7f50948a0a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:51.796 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.19.10, , , , , ] 2024-05-26T20:35:51.796 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:35:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:35:56.797+0000 7f50948a0a80 1 waiting for incremental sync to catch up: 2024-05-26T20:35:56.797 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.19.10, , , , , ] 2024-05-26T20:35:56.797 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:36:01.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:36:01.798+0000 7f50948a0a80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:01.799 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000005.19.10, , , 00000000027.71.6, , ] 2024-05-26T20:36:01.799 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:36:06.800 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:36:06.799+0000 7f50948a0a80 1 bucket sync caught up with source: 2024-05-26T20:36:06.800 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000006.21.10, , , 00000000027.71.6, , ] 2024-05-26T20:36:06.800 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000020.50.12, , ] 2024-05-26T20:36:06.800 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:36:06.799+0000 7f50948a0a80 0 bucket checkpoint complete 2024-05-26T20:36:06.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:06.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:06.816 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:36:06.883 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:36:06.883 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:36:06.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:06.923+0000 7f850e7eaa80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:06.923 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000005.19.10, , , 00000000026.68.6, , ] 2024-05-26T20:36:06.923 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000044.116.10, , ] 2024-05-26T20:36:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:11.924+0000 7f850e7eaa80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:11.924 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000006.21.10, , , 00000000026.68.6, , ] 2024-05-26T20:36:11.925 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000044.116.10, , ] 2024-05-26T20:36:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:16.925+0000 7f850e7eaa80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:16.926 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000006.21.10, , , 00000000026.68.6, , ] 2024-05-26T20:36:16.926 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000044.116.10, , ] 2024-05-26T20:36:21.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:21.927+0000 7f850e7eaa80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:21.927 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000006.21.10, , , 00000000026.68.6, , ] 2024-05-26T20:36:21.927 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000044.116.10, , ] 2024-05-26T20:36:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:26.928+0000 7f850e7eaa80 1 bucket sync caught up with source: 2024-05-26T20:36:26.928 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000006.21.10, , , 00000000045.118.10, , ] 2024-05-26T20:36:26.928 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000006.21.10, , , 00000000044.116.10, , ] 2024-05-26T20:36:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:26.928+0000 7f850e7eaa80 0 bucket checkpoint complete 2024-05-26T20:36:26.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:26.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:26.945 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:36:27.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:36:27.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:36:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:27.048+0000 7f7476754a80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:27.048 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000042.147.12, , ] 2024-05-26T20:36:27.048 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:32.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:32.050+0000 7f7476754a80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:32.050 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000042.147.12, , ] 2024-05-26T20:36:32.050 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:37.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:37.051+0000 7f7476754a80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:37.051 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000042.147.12, , ] 2024-05-26T20:36:37.051 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:42.052+0000 7f7476754a80 1 bucket sync caught up with source: 2024-05-26T20:36:42.052 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:42.052 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:42.052+0000 7f7476754a80 0 bucket checkpoint complete 2024-05-26T20:36:42.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:42.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:42.069 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:36:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:36:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:36:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:42.173+0000 7f5e983f6a80 1 waiting for incremental sync to catch up: 2024-05-26T20:36:42.173 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000044.151.10, , ] 2024-05-26T20:36:42.173 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000045.153.10, , ] 2024-05-26T20:36:47.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:47.174+0000 7f5e983f6a80 1 bucket sync caught up with source: 2024-05-26T20:36:47.174 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000045.153.10, , ] 2024-05-26T20:36:47.174 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000045.153.10, , ] 2024-05-26T20:36:47.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:47.174+0000 7f5e983f6a80 0 bucket checkpoint complete 2024-05-26T20:36:47.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:47.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:47.191 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:36:47.254 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:36:47.255 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:36:47.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:47.293+0000 7fe2e309ea80 1 bucket sync caught up with source: 2024-05-26T20:36:47.293 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000004.118.10, , , 00000000044.213.10, , ] 2024-05-26T20:36:47.293 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000004.118.10, , , 00000000044.213.10, , ] 2024-05-26T20:36:47.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:36:47.293+0000 7fe2e309ea80 0 bucket checkpoint complete 2024-05-26T20:36:47.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:47.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:36:47.306 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:36:47.374 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:36:47.374 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:36:47.412 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:36:47.412+0000 7fb3938b3a80 1 bucket sync caught up with source: 2024-05-26T20:36:47.412 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , 00000000004.118.10, , , 00000000044.213.10, , ] 2024-05-26T20:36:47.412 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , 00000000004.118.10, , , 00000000044.213.10, , ] 2024-05-26T20:36:47.413 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:36:47.412+0000 7fb3938b3a80 0 bucket checkpoint complete 2024-05-26T20:36:47.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone2, test-zone1} 2024-05-26T20:36:47.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.441 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:47.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:47.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:47.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:47.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:48.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:49.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:50.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone2, test-zone1} 2024-05-26T20:36:51.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone3, test-zone1} 2024-05-26T20:36:51.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.647 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:51.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:51.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:51.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:51.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:51.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:52.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:53.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:54.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:55.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone3, test-zone1} 2024-05-26T20:36:56.594 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-22 zones={test-zone3, test-zone2} 2024-05-26T20:36:56.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:36:56.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:56.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.605 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:36:56.612 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-05-26T20:36:56.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:36:56.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:56.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:56.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-05-26T20:36:56.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:56.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:57.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:58.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:36:59.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:00.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:01.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:02.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:03.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:04.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:04.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:04.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:04.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-22 zones={test-zone3, test-zone2} 2024-05-26T20:37:04.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.218 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.273 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.273 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.322 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:04.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.322 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-05-26T20:37:04.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:31:57.411978Z" 2024-05-26T20:37:04.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:36:21.428610Z" 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007844", 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:35:59.426801Z" 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:26:27.404806Z" 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.323 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:04.324 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T20:37:04.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.325 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.384 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.384 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.431 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 16224333313614393590 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:37:04.432 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:31:57.411978Z" 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.433 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:36:21.428610Z" 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000007844", 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:04.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:35:59.426801Z" 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:26:27.404806Z" 2024-05-26T20:37:04.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 0, 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 0 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:37:04Z" 2024-05-26T20:37:04.436 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:37:04.437 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T20:37:04.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.438 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:31:57.411978Z" 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:36:21.428610Z" 2024-05-26T20:37:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007844", 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:35:59.426801Z" 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:26:27.404806Z" 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.536 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:04.537 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-05-26T20:37:04.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.538 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.589 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.589 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 11991635280324976904 2024-05-26T20:37:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:31:57.411978Z" 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:36:21.428610Z" 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007844", 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:35:59.426801Z" 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.635 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:26:27.404806Z" 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.636 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:04Z" 2024-05-26T20:37:04.637 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:04.638 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-05-26T20:37:04.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.638 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout:[ 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:33:45.642901Z" 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:36:43.498741Z" 2024-05-26T20:37:04.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:35:15.493189Z" 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:26:05.478477Z" 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.737 INFO:teuthology.orchestra.run.smithi137.stdout:] 2024-05-26T20:37:04.739 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-05-26T20:37:04.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.739 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.834 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.835 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:33:45.642901Z" 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:36:43.498741Z" 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:35:15.493189Z" 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:04.837 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:26:05.478477Z" 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T20:37:04.838 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T20:37:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:04Z" 2024-05-26T20:37:04.839 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:04.840 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-05-26T20:37:04.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.840 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.889 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:04.933 INFO:teuthology.orchestra.run.smithi137.stdout:[ 2024-05-26T20:37:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:33:45.642901Z" 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:36:43.498741Z" 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:35:15.493189Z" 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T20:26:05.478477Z" 2024-05-26T20:37:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:04.935 INFO:teuthology.orchestra.run.smithi137.stdout:] 2024-05-26T20:37:04.935 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-05-26T20:37:04.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:04.936 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:04.989 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:04.989 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.031 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 15656959347356121910 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:33:45.642901Z" 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.033 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:36:43.498741Z" 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-05-26T20:37:05.034 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:35:15.493189Z" 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:26:05.478477Z" 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T20:37:05.036 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T20:37:05.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.037 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:05Z" 2024-05-26T20:37:05.037 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:05.038 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-05-26T20:37:05.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.038 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.089 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.136 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:32:03.537052Z" 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:05.137 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:36:27.551514Z" 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007985", 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:35:21.546487Z" 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:26:11.530185Z" 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.138 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:05.140 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-05-26T20:37:05.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.140 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.193 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.193 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 13112891809682280895 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:32:03.537052Z" 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:36:27.551514Z" 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007985", 2024-05-26T20:37:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:35:21.546487Z" 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:26:11.530185Z" 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:05Z" 2024-05-26T20:37:05.239 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:05.240 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-05-26T20:37:05.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.241 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.290 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.290 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.336 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:05.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.336 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:32:03.537052Z" 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:36:27.551514Z" 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007985", 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:35:21.546487Z" 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.338 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:05.338 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:26:11.530185Z" 2024-05-26T20:37:05.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.338 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:05.339 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-05-26T20:37:05.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.339 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.394 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 2930100838285975777 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:37:05.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:32:03.537052Z" 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:37:05.436 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:36:27.551514Z" 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.437 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000007985", 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:35:21.546487Z" 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-05-26T20:37:05.438 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:26:11.530185Z" 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 0, 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 0 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:37:05Z" 2024-05-26T20:37:05.439 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:37:05.441 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-05-26T20:37:05.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-05-26T20:37:05.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.455 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.535 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:37:05.547 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:05.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.547 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755825.454470_454.1", 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:37:05.454470Z" 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:37:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.549 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:05.550 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:37:05.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.550 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.602 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.603 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.630 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:05.630+0000 7f15cffd3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:05.630 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:05.630+0000 7f15cffd3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:05.642 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:37:05.642 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755825.454470_454.1", 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:37:05.454470Z", 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:37:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:37:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:37:05Z" 2024-05-26T20:37:05.650 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:37:05.651 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:37:05.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:37:05.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:05.651 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:05.702 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:05.702 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:05.731+0000 7f7e1bf7ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:05.732+0000 7f7e1bf7ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:05.742 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:05.742 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:05.742 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:37:05.743 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.744 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755825.448745_451.1", 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:37:05.448745Z", 2024-05-26T20:37:05.745 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:05.746 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:05Z" 2024-05-26T20:37:05.747 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:05.748 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716755825.454470_454.1 target=1_1716755825.448745_451.1 2024-05-26T20:37:10.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:10.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:10.750 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:10.809 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:10.809 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:10.841+0000 7fd6a2c65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:10.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:10.842+0000 7fd6a2c65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:10.855 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:10.856 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:10.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755825.454470_454.1", 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:37:05.454470Z", 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:10.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:10.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:10Z" 2024-05-26T20:37:10.860 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:10.861 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:37:10.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:10.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:10.870 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:10.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:10.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:10.969 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:10.969+0000 7f6db30eda80 1 waiting to reach incremental sync.. 2024-05-26T20:37:15.969 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:15.970+0000 7f6db30eda80 1 waiting to reach incremental sync.. 2024-05-26T20:37:20.971 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:20.972+0000 7f6db30eda80 1 bucket sync caught up with source: 2024-05-26T20:37:20.972 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:20.972 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:20.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:20.972+0000 7f6db30eda80 0 bucket checkpoint complete 2024-05-26T20:37:20.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:20.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:20.988 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:21.054 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:21.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:21.095+0000 7f938ff7ca80 1 waiting to reach incremental sync.. 2024-05-26T20:37:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.097+0000 7f938ff7ca80 1 bucket sync caught up with source: 2024-05-26T20:37:26.097 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:26.097 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.097+0000 7f938ff7ca80 0 bucket checkpoint complete 2024-05-26T20:37:26.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.113 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:26.177 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:26.177 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:26.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.217+0000 7f55e0d95a80 1 bucket sync caught up with source: 2024-05-26T20:37:26.217 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:37:26.217 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:37:26.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.217+0000 7f55e0d95a80 0 bucket checkpoint complete 2024-05-26T20:37:26.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.230 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:26.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.324+0000 7fd7c2f00a80 1 bucket sync caught up with source: 2024-05-26T20:37:26.324 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:37:26.324 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:37:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.324+0000 7fd7c2f00a80 0 bucket checkpoint complete 2024-05-26T20:37:26.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:26.337 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:26.428+0000 7f6e5beafa80 1 waiting to reach incremental sync.. 2024-05-26T20:37:31.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:31.429+0000 7f6e5beafa80 1 waiting to reach incremental sync.. 2024-05-26T20:37:36.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:36.430+0000 7f6e5beafa80 1 waiting to reach incremental sync.. 2024-05-26T20:37:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:41.433+0000 7f6e5beafa80 1 bucket sync caught up with source: 2024-05-26T20:37:41.434 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-05-26T20:37:41.434 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.29.6, , ] 2024-05-26T20:37:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:41.433+0000 7f6e5beafa80 0 bucket checkpoint complete 2024-05-26T20:37:41.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:41.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:41.450 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:41.511 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:41.511 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:41.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:41.548+0000 7fb175143a80 1 bucket sync caught up with source: 2024-05-26T20:37:41.548 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-05-26T20:37:41.548 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.29.6, , ] 2024-05-26T20:37:41.548 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:41.548+0000 7fb175143a80 0 bucket checkpoint complete 2024-05-26T20:37:41.561 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:37:41.566 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:37:41.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.569 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:37:41.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:37:41.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:37:41.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:37:41.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.697 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:37:41.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:37:41.821 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:37:41.826 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:37:41.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.828 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:37:41.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:37:41.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:37:41.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:37:41.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:41.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:41.959 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:42.017 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:42.017 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:42.051 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:42.064 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755861.959709_474.1", 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:37:41.959709Z" 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.065 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:37:42.066 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:37:42.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:42.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:42.067 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:42.117 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:42.117 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:42.144+0000 7f777a39da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:42.145+0000 7f777a39da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:42.157 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:37:42.157 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:37:42.157 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:37:42.157 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:42.158 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:37:42.159 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755861.959709_474.1", 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:37:41.959709Z", 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:42.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:42.162 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:37:42Z" 2024-05-26T20:37:42.163 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:37:42.164 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:37:42.165 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:37:42.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:42.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:42.165 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:42.247+0000 7ff820a5fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:42.248+0000 7ff820a5fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:42.259 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:42.260 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:42.261 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:42.262 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:37:42.263 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:42.264 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755825.454470_454.1", 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:37:05.454470Z", 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:42.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:42Z" 2024-05-26T20:37:42.267 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:42.268 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716755861.959709_474.1 target=1_1716755825.454470_454.1 2024-05-26T20:37:47.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:47.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:37:47.270 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:37:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:47.328 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:47.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:47.361+0000 7f88d3314a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:47.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:37:47.362+0000 7f88d3314a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:47.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:37:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755861.959709_474.1", 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:37:41.959709Z", 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:37:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:37:47.379 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:37:47.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:37:47.380 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:37:47Z" 2024-05-26T20:37:47.380 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:37:47.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:37:47.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:47.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:37:47.429 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:37:47.494 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:37:47.494 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:37:47.532 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:47.532+0000 7fe9f6101a80 1 waiting for incremental sync to catch up: 2024-05-26T20:37:47.532 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:47.532 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:37:52.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:52.533+0000 7fe9f6101a80 1 waiting for incremental sync to catch up: 2024-05-26T20:37:52.534 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:52.534 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:37:57.535 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:37:57.534+0000 7fe9f6101a80 1 waiting for incremental sync to catch up: 2024-05-26T20:37:57.535 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:37:57.535 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:02.536 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:02.536+0000 7fe9f6101a80 1 bucket sync caught up with source: 2024-05-26T20:38:02.536 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:02.536 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:02.536 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:02.536+0000 7fe9f6101a80 0 bucket checkpoint complete 2024-05-26T20:38:02.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:02.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:02.552 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:02.618 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:02.618 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:02.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:02.658+0000 7f1ac0285a80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:02.658 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2024-05-26T20:38:02.658 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:07.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:07.665+0000 7f1ac0285a80 1 bucket sync caught up with source: 2024-05-26T20:38:07.665 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:07.665 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:07.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:07.665+0000 7f1ac0285a80 0 bucket checkpoint complete 2024-05-26T20:38:07.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:07.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:07.731 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:07.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:07.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:07.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:07.827+0000 7efe99cd6a80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:07.827 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:38:07.827 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:12.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:12.828+0000 7efe99cd6a80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:12.828 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:38:12.828 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:17.829+0000 7efe99cd6a80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:17.830 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2024-05-26T20:38:17.830 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:22.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:22.831+0000 7efe99cd6a80 1 bucket sync caught up with source: 2024-05-26T20:38:22.831 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:22.831 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:22.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:22.831+0000 7efe99cd6a80 0 bucket checkpoint complete 2024-05-26T20:38:22.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:22.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:22.848 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:22.910 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:22.950+0000 7fb66ce6ca80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:22.950 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.29.13, , ] 2024-05-26T20:38:22.950 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000004.31.10, , ] 2024-05-26T20:38:27.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:27.951+0000 7fb66ce6ca80 1 bucket sync caught up with source: 2024-05-26T20:38:27.951 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.31.10, , ] 2024-05-26T20:38:27.951 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000004.31.10, , ] 2024-05-26T20:38:27.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:27.951+0000 7fb66ce6ca80 0 bucket checkpoint complete 2024-05-26T20:38:27.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:27.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:27.968 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:28.031 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:28.031 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:28.069+0000 7f3c397fca80 1 bucket sync caught up with source: 2024-05-26T20:38:28.069 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:28.070 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:28.069+0000 7f3c397fca80 0 bucket checkpoint complete 2024-05-26T20:38:28.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:28.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:28.083 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:28.147 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:28.147 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:28.185 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:28.185+0000 7ff749c18a80 1 bucket sync caught up with source: 2024-05-26T20:38:28.185 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:28.185 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:28.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:28.185+0000 7ff749c18a80 0 bucket checkpoint complete 2024-05-26T20:38:28.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:38:28.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:38:28.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:38:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:38:28.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:38:28.409 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:38:28.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.411 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:38:28.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:38:28.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:38:28.607 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:38:28.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.610 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:38:28.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:38:28.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:38:28.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:38:28.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:28.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:28.803 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:38:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:28.886 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:28.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755908.802972_484.1", 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:38:28.802972Z" 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:28.899 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:38:28.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:38:28.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:28.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:28.901 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:38:28.955 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:28.955 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:28.984 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:28.984+0000 7fc9957a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:28.985 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:28.985+0000 7fc9957a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:28.997 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:38:28.998 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:28.999 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:29.000 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755861.959709_474.1", 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:37:41.959709Z", 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:29.001 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:38:29.002 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:38:29.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:29.003 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:38:28Z" 2024-05-26T20:38:29.003 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:38:29.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716755908.802972_484.1 target=1_1716755861.959709_474.1 2024-05-26T20:38:34.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:34.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:34.005 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:38:34.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:34.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:34.092+0000 7f1649b35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:34.093+0000 7f1649b35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:34.106 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:38:34.107 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:34.108 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755908.802972_484.1", 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:38:28.802972Z", 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:38:34.109 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.110 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:38:34.111 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:38:34.111 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:38:34.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:38:34.111 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:38:34Z" 2024-05-26T20:38:34.111 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:38:34.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:38:34.112 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:38:34.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:34.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:38:34.112 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:38:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:34.206+0000 7f749aca0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:34.207+0000 7f749aca0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:38:34.219 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:38:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:34.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755908.802972_484.1", 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:38:28.802972Z", 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.223 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:38:34.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:38:34.224 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:38:34.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:38:34.224 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:38:34Z" 2024-05-26T20:38:34.224 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:38:34.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:38:34.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:34.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:34.239 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:34.332 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:34.332+0000 7f13d42dda80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:34.332 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:34.332 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:39.334 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:39.333+0000 7f13d42dda80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:39.334 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:39.334 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:44.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:44.335+0000 7f13d42dda80 1 bucket sync caught up with source: 2024-05-26T20:38:44.335 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:44.336 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:44.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:38:44.335+0000 7f13d42dda80 0 bucket checkpoint complete 2024-05-26T20:38:44.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:44.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:44.352 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:44.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:44.456+0000 7fdc83fe5a80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:44.456 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.20.13, , ] 2024-05-26T20:38:44.457 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:49.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.458+0000 7fdc83fe5a80 1 bucket sync caught up with source: 2024-05-26T20:38:49.458 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:49.458 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000005.24.18, , ] 2024-05-26T20:38:49.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.458+0000 7fdc83fe5a80 0 bucket checkpoint complete 2024-05-26T20:38:49.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.474 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:49.538 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.577+0000 7f595644da80 1 bucket sync caught up with source: 2024-05-26T20:38:49.577 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000006.37.18, , ] 2024-05-26T20:38:49.577 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000006.37.18, , ] 2024-05-26T20:38:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.577+0000 7f595644da80 0 bucket checkpoint complete 2024-05-26T20:38:49.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.591 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:49.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.685+0000 7f936bff4a80 1 bucket sync caught up with source: 2024-05-26T20:38:49.685 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000006.37.18, , ] 2024-05-26T20:38:49.685 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000006.37.18, , ] 2024-05-26T20:38:49.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.685+0000 7f936bff4a80 0 bucket checkpoint complete 2024-05-26T20:38:49.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:38:49.697 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:38:49.755 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:38:49.755 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:38:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:49.790+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:49.790 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:49.790 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:38:54.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:54.791+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:54.791 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:54.791 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:38:59.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:38:59.792+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:38:59.793 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:38:59.793 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:04.794+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:39:04.797 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:39:04.797 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:09.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:09.795+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:39:09.796 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:39:09.796 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:14.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:14.796+0000 7f96f4d0da80 1 waiting for incremental sync to catch up: 2024-05-26T20:39:14.797 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:39:14.797 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:19.798+0000 7f96f4d0da80 1 bucket sync caught up with source: 2024-05-26T20:39:19.798 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:19.798 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:19.798+0000 7f96f4d0da80 0 bucket checkpoint complete 2024-05-26T20:39:19.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:19.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:19.814 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:19.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:19.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:19.917+0000 7fd01a12fa80 1 waiting for incremental sync to catch up: 2024-05-26T20:39:19.916 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000004.38.10, , ] 2024-05-26T20:39:19.916 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:24.918 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:24.917+0000 7fd01a12fa80 1 bucket sync caught up with source: 2024-05-26T20:39:24.918 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:24.918 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000009.50.15, , ] 2024-05-26T20:39:24.918 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:24.917+0000 7fd01a12fa80 0 bucket checkpoint complete 2024-05-26T20:39:24.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:39:24.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:39:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:24.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.139 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone2, test-zone1} 2024-05-26T20:39:25.139 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:39:25.144 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:39:25.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.147 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:39:25.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.299 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone1} 2024-05-26T20:39:25.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:39:25.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:39:25.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:39:25.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:39:25.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:39:25.595 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-23 zones={test-zone3, test-zone2} 2024-05-26T20:39:25.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-05-26T20:39:25.596 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-24 2024-05-26T20:39:25.612 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-25 2024-05-26T20:39:25.641 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-26 2024-05-26T20:39:25.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:25.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:25.690 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:39:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:25.742 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:39:25.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755965.688474_523.1", 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:39:25.688474Z" 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:25.787 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:39:25.788 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:39:25.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:25.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:25.789 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:39:25.846 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:25.846 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:25.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:25.882+0000 7fc27e729a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:25.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:25.883+0000 7fc27e729a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:25.895 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:39:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:39:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:39:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:39:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:39:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755965.623102_510.1", 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:39:25.623102Z", 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:39:25Z" 2024-05-26T20:39:25.900 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:39:25.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716755965.688474_523.1 target=1_1716755965.623102_510.1 2024-05-26T20:39:30.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:30.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:30.903 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:39:30.960 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:30.961 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:30.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:30.991+0000 7f8c44816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:30.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:30.992+0000 7f8c44816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:31.004 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:39:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:39:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:39:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:39:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716755965.688474_523.1", 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:39:25.688474Z", 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:39:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:39:30Z" 2024-05-26T20:39:31.010 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:39:31.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:39:31.012 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:39:31.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:31.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:39:31.012 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:39:31.070 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:31.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:31.106+0000 7f6cf3936a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:31.107+0000 7f6cf3936a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:39:31.119 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:39:31.119 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.120 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:39:31.121 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:39:31.122 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:31.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716755965.688474_523.1", 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:39:31.124 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:39:25.688474Z", 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:39:31.125 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:39:31.126 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:39:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:39:31Z" 2024-05-26T20:39:31.128 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:39:31.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:39:31.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:31.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:31.195 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:31.251 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:31.251 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:31.285 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:31.285+0000 7f2716ebfa80 1 waiting to reach incremental sync.. 2024-05-26T20:39:36.286 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:36.285+0000 7f2716ebfa80 1 waiting to reach incremental sync.. 2024-05-26T20:39:41.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:41.287+0000 7f2716ebfa80 1 bucket sync caught up with source: 2024-05-26T20:39:41.287 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.49.12, , ] 2024-05-26T20:39:41.287 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.49.12, , ] 2024-05-26T20:39:41.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:39:41.287+0000 7f2716ebfa80 0 bucket checkpoint complete 2024-05-26T20:39:41.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:41.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:41.300 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:41.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:41.448+0000 7ff83c19ba80 1 waiting to reach incremental sync.. 2024-05-26T20:39:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:46.451+0000 7ff83c19ba80 1 waiting to reach incremental sync.. 2024-05-26T20:39:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.453+0000 7ff83c19ba80 1 bucket sync caught up with source: 2024-05-26T20:39:51.453 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.49.12, , ] 2024-05-26T20:39:51.453 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.49.12, , ] 2024-05-26T20:39:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.453+0000 7ff83c19ba80 0 bucket checkpoint complete 2024-05-26T20:39:51.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.470 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:51.533 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:51.533 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.572+0000 7f651a5c3a80 1 bucket sync caught up with source: 2024-05-26T20:39:51.572 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.92.6, , ] 2024-05-26T20:39:51.572 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.92.6, , ] 2024-05-26T20:39:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.572+0000 7f651a5c3a80 0 bucket checkpoint complete 2024-05-26T20:39:51.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.646 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.743+0000 7fa20d786a80 1 bucket sync caught up with source: 2024-05-26T20:39:51.743 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.92.6, , ] 2024-05-26T20:39:51.743 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.92.6, , ] 2024-05-26T20:39:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.743+0000 7fa20d786a80 0 bucket checkpoint complete 2024-05-26T20:39:51.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:39:51.756 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:39:51.812 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:39:51.813 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:39:51.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:51.847+0000 7f90f5a5da80 1 waiting to reach incremental sync.. 2024-05-26T20:39:56.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:39:56.848+0000 7f90f5a5da80 1 waiting to reach incremental sync.. 2024-05-26T20:40:01.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:01.849+0000 7f90f5a5da80 1 bucket sync caught up with source: 2024-05-26T20:40:01.849 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.165.6, , ] 2024-05-26T20:40:01.849 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.165.6, , ] 2024-05-26T20:40:01.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:01.849+0000 7f90f5a5da80 0 bucket checkpoint complete 2024-05-26T20:40:01.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:01.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:01.866 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:01.928 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:01.928 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:01.964 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:01.965+0000 7f4f9f506a80 1 bucket sync caught up with source: 2024-05-26T20:40:01.964 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000002.165.6, , ] 2024-05-26T20:40:01.964 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.165.6, , ] 2024-05-26T20:40:01.965 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:01.965+0000 7f4f9f506a80 0 bucket checkpoint complete 2024-05-26T20:40:01.977 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-24 zones={test-zone2, test-zone1} 2024-05-26T20:40:01.982 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:01.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:01.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:01.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:01.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:01.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:01.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-24 zones={test-zone2, test-zone1} 2024-05-26T20:40:02.015 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-24 zones={test-zone3, test-zone1} 2024-05-26T20:40:02.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:02.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.022 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:02.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-24 zones={test-zone3, test-zone1} 2024-05-26T20:40:02.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-24 zones={test-zone3, test-zone2} 2024-05-26T20:40:02.146 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:02.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.191 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:02.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-24 zones={test-zone3, test-zone2} 2024-05-26T20:40:02.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.263 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.354 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:02.354+0000 7eff4a8d8a80 1 bucket sync caught up with source: 2024-05-26T20:40:02.354 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000002.30.6, , ] 2024-05-26T20:40:02.354 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.30.6, , ] 2024-05-26T20:40:02.354 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:02.354+0000 7eff4a8d8a80 0 bucket checkpoint complete 2024-05-26T20:40:02.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.366 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.430 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.430 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.470+0000 7f3daee4da80 1 bucket sync caught up with source: 2024-05-26T20:40:02.470 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.30.6, , ] 2024-05-26T20:40:02.470 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.30.6, , ] 2024-05-26T20:40:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.470+0000 7f3daee4da80 0 bucket checkpoint complete 2024-05-26T20:40:02.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.483 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.576+0000 7fe79f3ffa80 1 bucket sync caught up with source: 2024-05-26T20:40:02.576 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.45.12, , ] 2024-05-26T20:40:02.576 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.45.12, , ] 2024-05-26T20:40:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.576+0000 7fe79f3ffa80 0 bucket checkpoint complete 2024-05-26T20:40:02.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.588 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.645 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.680+0000 7fc102150a80 1 bucket sync caught up with source: 2024-05-26T20:40:02.681 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.45.12, , ] 2024-05-26T20:40:02.681 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.45.12, , ] 2024-05-26T20:40:02.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.680+0000 7fc102150a80 0 bucket checkpoint complete 2024-05-26T20:40:02.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.693 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.749 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.750 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.786+0000 7f9ffbd24a80 1 bucket sync caught up with source: 2024-05-26T20:40:02.786 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.42.6, , ] 2024-05-26T20:40:02.786 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.42.6, , ] 2024-05-26T20:40:02.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:02.786+0000 7f9ffbd24a80 0 bucket checkpoint complete 2024-05-26T20:40:02.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:02.798 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:02.857 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:02.857 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:02.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:02.893+0000 7fdfaf03da80 1 bucket sync caught up with source: 2024-05-26T20:40:02.892 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000002.42.6, , ] 2024-05-26T20:40:02.892 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.42.6, , ] 2024-05-26T20:40:02.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:02.893+0000 7fdfaf03da80 0 bucket checkpoint complete 2024-05-26T20:40:02.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-25 zones={test-zone2, test-zone1} 2024-05-26T20:40:02.909 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:02.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:02.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:02.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:02.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-25 zones={test-zone2, test-zone1} 2024-05-26T20:40:03.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-25 zones={test-zone3, test-zone1} 2024-05-26T20:40:03.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:03.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.079 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:03.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-25 zones={test-zone3, test-zone1} 2024-05-26T20:40:03.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-25 zones={test-zone3, test-zone2} 2024-05-26T20:40:03.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:03.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:03.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:03.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:03.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-25 zones={test-zone3, test-zone2} 2024-05-26T20:40:03.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:03.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:03.279 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:03.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:03.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:03.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:03.371+0000 7fe63ee96a80 1 waiting for incremental sync to catch up: 2024-05-26T20:40:03.370 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.7.11, , ] 2024-05-26T20:40:03.370 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:08.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:08.372+0000 7fe63ee96a80 1 waiting for incremental sync to catch up: 2024-05-26T20:40:08.372 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.7.11, , ] 2024-05-26T20:40:08.372 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:13.373 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:13.373+0000 7fe63ee96a80 1 waiting for incremental sync to catch up: 2024-05-26T20:40:13.373 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.7.11, , ] 2024-05-26T20:40:13.373 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:18.374 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:18.374+0000 7fe63ee96a80 1 waiting for incremental sync to catch up: 2024-05-26T20:40:18.374 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.7.11, , ] 2024-05-26T20:40:18.374 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:23.375 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:23.375+0000 7fe63ee96a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.375 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:23.376 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:23.376 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:23.375+0000 7fe63ee96a80 0 bucket checkpoint complete 2024-05-26T20:40:23.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.392 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:23.458 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:23.458 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.497+0000 7f911a194a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.497 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:23.497 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.12.6, , ] 2024-05-26T20:40:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.497+0000 7f911a194a80 0 bucket checkpoint complete 2024-05-26T20:40:23.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.510 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:23.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:23.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.604+0000 7f09acb35a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.604 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.20.6, , ] 2024-05-26T20:40:23.604 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.20.6, , ] 2024-05-26T20:40:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.604+0000 7f09acb35a80 0 bucket checkpoint complete 2024-05-26T20:40:23.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.616 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:23.674 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:23.674 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.709+0000 7fae795e6a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.709 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000002.20.6, , ] 2024-05-26T20:40:23.709 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000002.20.6, , ] 2024-05-26T20:40:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.709+0000 7fae795e6a80 0 bucket checkpoint complete 2024-05-26T20:40:23.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.722 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:23.779 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:23.779 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:23.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.813+0000 7f3ba2627a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.814 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.22.12, , ] 2024-05-26T20:40:23.814 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2024-05-26T20:40:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:23.813+0000 7f3ba2627a80 0 bucket checkpoint complete 2024-05-26T20:40:23.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:23.826 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:23.890 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:23.890 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:23.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:23.927+0000 7f91756f5a80 1 bucket sync caught up with source: 2024-05-26T20:40:23.927 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.22.12, , ] 2024-05-26T20:40:23.927 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2024-05-26T20:40:23.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:23.927+0000 7f91756f5a80 0 bucket checkpoint complete 2024-05-26T20:40:23.941 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-26 zones={test-zone2, test-zone1} 2024-05-26T20:40:23.946 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:23.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:23.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:23.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:23.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:23.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:23.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:23.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:23.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-26 zones={test-zone2, test-zone1} 2024-05-26T20:40:24.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-26 zones={test-zone3, test-zone1} 2024-05-26T20:40:24.025 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:24.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:24.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.148 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-26 zones={test-zone3, test-zone1} 2024-05-26T20:40:24.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-26 zones={test-zone3, test-zone2} 2024-05-26T20:40:24.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:40:24.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:40:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:40:24.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:40:24.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-26 zones={test-zone3, test-zone2} 2024-05-26T20:40:24.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-05-26T20:40:24.295 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-27 2024-05-26T20:40:24.308 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-28 2024-05-26T20:40:24.330 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-29 2024-05-26T20:40:24.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:24.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:24.389 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:40:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:24.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:40:24.510 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:24.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756024.387658_576.1", 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:40:24.387658Z" 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:24.512 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:40:24.513 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:40:24.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:24.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:24.514 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:40:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:24.592 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:24.591+0000 7f7fc5caaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:24.592 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:24.592+0000 7f7fc5caaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:24.604 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:40:24.604 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:40:24.604 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:40:24.605 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:24.606 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756024.354368_565.1", 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:40:24.354368Z", 2024-05-26T20:40:24.607 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:24.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:40:24Z" 2024-05-26T20:40:24.609 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:40:24.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756024.387658_576.1 target=1_1716756024.354368_565.1 2024-05-26T20:40:29.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:29.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:29.611 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:40:29.669 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:29.669 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:29.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:29.700+0000 7fef36c33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:29.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:29.702+0000 7fef36c33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:29.714 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:40:29.714 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:40:29.714 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:29.716 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756024.387658_576.1", 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:40:24.387658Z", 2024-05-26T20:40:29.717 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:40:29.718 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:40:29Z" 2024-05-26T20:40:29.719 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:40:29.720 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:40:29.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:40:29.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:29.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:40:29.721 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:40:29.779 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:29.780 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:29.812+0000 7fb250acca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:29.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:29.813+0000 7fb250acca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:40:29.825 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:40:29.825 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:40:29.825 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:40:29.825 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:40:29.826 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:40:29.827 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756024.387658_576.1", 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:40:24.387658Z", 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:40:29.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:29.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:40:29Z" 2024-05-26T20:40:29.830 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:40:29.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T20:40:29.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:29.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:29.931 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:29.988 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:29.988 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:30.021 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:30.021+0000 7fa643414a80 1 waiting to reach incremental sync.. 2024-05-26T20:40:35.022 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:35.022+0000 7fa643414a80 1 waiting to reach incremental sync.. 2024-05-26T20:40:40.023 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:40.022+0000 7fa643414a80 1 waiting to reach incremental sync.. 2024-05-26T20:40:45.025 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:45.024+0000 7fa643414a80 1 bucket sync caught up with source: 2024-05-26T20:40:45.025 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.22.7, , ] 2024-05-26T20:40:45.025 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.22.7, , ] 2024-05-26T20:40:45.025 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:40:45.024+0000 7fa643414a80 0 bucket checkpoint complete 2024-05-26T20:40:45.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:45.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:45.041 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:45.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:45.146+0000 7f99c3046a80 1 waiting to reach incremental sync.. 2024-05-26T20:40:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.148+0000 7f99c3046a80 1 bucket sync caught up with source: 2024-05-26T20:40:50.148 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.22.7, , ] 2024-05-26T20:40:50.148 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.22.7, , ] 2024-05-26T20:40:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.148+0000 7f99c3046a80 0 bucket checkpoint complete 2024-05-26T20:40:50.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.165 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:50.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:50.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:50.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.270+0000 7ffa49b77a80 1 bucket sync caught up with source: 2024-05-26T20:40:50.270 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.23.14, , ] 2024-05-26T20:40:50.270 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.23.14, , ] 2024-05-26T20:40:50.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.270+0000 7ffa49b77a80 0 bucket checkpoint complete 2024-05-26T20:40:50.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.283 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:50.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.378+0000 7f1789c20a80 1 bucket sync caught up with source: 2024-05-26T20:40:50.378 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.23.14, , ] 2024-05-26T20:40:50.378 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.23.14, , ] 2024-05-26T20:40:50.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.378+0000 7f1789c20a80 0 bucket checkpoint complete 2024-05-26T20:40:50.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:40:50.391 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:40:50.447 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:40:50.448 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:40:50.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:50.482+0000 7f91ee94ba80 1 waiting to reach incremental sync.. 2024-05-26T20:40:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:40:55.483+0000 7f91ee94ba80 1 waiting to reach incremental sync.. 2024-05-26T20:41:00.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:00.484+0000 7f91ee94ba80 1 bucket sync caught up with source: 2024-05-26T20:41:00.485 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.27.14, , ] 2024-05-26T20:41:00.485 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.27.14, , ] 2024-05-26T20:41:00.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:00.484+0000 7f91ee94ba80 0 bucket checkpoint complete 2024-05-26T20:41:00.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:00.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:00.501 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:00.565 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:00.565 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:00.602 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:00.603+0000 7f47ae3d0a80 1 waiting to reach incremental sync.. 2024-05-26T20:41:05.604 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:05.605+0000 7f47ae3d0a80 1 bucket sync caught up with source: 2024-05-26T20:41:05.605 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.27.14, , ] 2024-05-26T20:41:05.605 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.27.14, , ] 2024-05-26T20:41:05.605 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:05.605+0000 7f47ae3d0a80 0 bucket checkpoint complete 2024-05-26T20:41:05.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-27 zones={test-zone2, test-zone1} 2024-05-26T20:41:05.626 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:05.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:05.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:05.682 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-27 zones={test-zone2, test-zone1} 2024-05-26T20:41:05.682 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-27 zones={test-zone3, test-zone1} 2024-05-26T20:41:05.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:05.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.689 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:05.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:05.744 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-27 zones={test-zone3, test-zone1} 2024-05-26T20:41:05.744 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-27 zones={test-zone3, test-zone2} 2024-05-26T20:41:05.749 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:05.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:05.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:05.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:05.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-27 zones={test-zone3, test-zone2} 2024-05-26T20:41:05.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.762 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:05.826 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:05.826 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:05.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:05.864+0000 7ff34984da80 1 bucket sync caught up with source: 2024-05-26T20:41:05.864 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.43.14, , ] 2024-05-26T20:41:05.864 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.43.14, , ] 2024-05-26T20:41:05.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:05.864+0000 7ff34984da80 0 bucket checkpoint complete 2024-05-26T20:41:05.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.878 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:05.943 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:05.943 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:05.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:05.983+0000 7ffa8c00ca80 1 bucket sync caught up with source: 2024-05-26T20:41:05.983 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.43.14, , ] 2024-05-26T20:41:05.983 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.43.14, , ] 2024-05-26T20:41:05.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:05.983+0000 7ffa8c00ca80 0 bucket checkpoint complete 2024-05-26T20:41:05.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:05.996 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.090+0000 7f784a621a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.090 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.83.7, , ] 2024-05-26T20:41:06.090 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.83.7, , ] 2024-05-26T20:41:06.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.090+0000 7f784a621a80 0 bucket checkpoint complete 2024-05-26T20:41:06.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.102 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.159 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.159 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.198+0000 7f0c88e38a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.198 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.83.7, , ] 2024-05-26T20:41:06.198 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.83.7, , ] 2024-05-26T20:41:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.198+0000 7f0c88e38a80 0 bucket checkpoint complete 2024-05-26T20:41:06.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.210 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.305+0000 7f7aeaab9a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.305 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.58.14, , ] 2024-05-26T20:41:06.305 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.58.14, , ] 2024-05-26T20:41:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.305+0000 7f7aeaab9a80 0 bucket checkpoint complete 2024-05-26T20:41:06.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.318 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.377 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.378 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.413 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:06.414+0000 7f1c5b72ba80 1 bucket sync caught up with source: 2024-05-26T20:41:06.414 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.58.14, , ] 2024-05-26T20:41:06.414 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.58.14, , ] 2024-05-26T20:41:06.414 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:06.414+0000 7f1c5b72ba80 0 bucket checkpoint complete 2024-05-26T20:41:06.426 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-28 zones={test-zone2, test-zone1} 2024-05-26T20:41:06.430 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:06.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:06.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:06.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-28 zones={test-zone2, test-zone1} 2024-05-26T20:41:06.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-28 zones={test-zone3, test-zone1} 2024-05-26T20:41:06.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:06.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.497 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:06.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:06.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-28 zones={test-zone3, test-zone1} 2024-05-26T20:41:06.506 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-28 zones={test-zone3, test-zone2} 2024-05-26T20:41:06.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:06.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.512 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:06.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:06.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:06.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-28 zones={test-zone3, test-zone2} 2024-05-26T20:41:06.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.523 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.578 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.578 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:06.614+0000 7f15412d3a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.613 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000001.35.14, , ] 2024-05-26T20:41:06.613 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000001.35.14, , ] 2024-05-26T20:41:06.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:06.614+0000 7f15412d3a80 0 bucket checkpoint complete 2024-05-26T20:41:06.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.625 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.717+0000 7effa0cbaa80 1 bucket sync caught up with source: 2024-05-26T20:41:06.717 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.35.14, , ] 2024-05-26T20:41:06.717 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.35.14, , ] 2024-05-26T20:41:06.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.717+0000 7effa0cbaa80 0 bucket checkpoint complete 2024-05-26T20:41:06.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.730 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.823+0000 7f68e0234a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.823 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.86.14, , ] 2024-05-26T20:41:06.823 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.86.14, , ] 2024-05-26T20:41:06.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.823+0000 7f68e0234a80 0 bucket checkpoint complete 2024-05-26T20:41:06.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.836 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:06.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.928+0000 7fd8a59c6a80 1 bucket sync caught up with source: 2024-05-26T20:41:06.928 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.86.14, , ] 2024-05-26T20:41:06.929 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.86.14, , ] 2024-05-26T20:41:06.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:06.928+0000 7fd8a59c6a80 0 bucket checkpoint complete 2024-05-26T20:41:06.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:06.941 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:06.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:06.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:07.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:07.033+0000 7f66fcc04a80 1 bucket sync caught up with source: 2024-05-26T20:41:07.033 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000003.212.7, , ] 2024-05-26T20:41:07.033 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000003.212.7, , ] 2024-05-26T20:41:07.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:07.033+0000 7f66fcc04a80 0 bucket checkpoint complete 2024-05-26T20:41:07.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:07.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T20:41:07.046 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T20:41:07.104 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:07.104 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:07.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:07.140+0000 7f1169698a80 1 bucket sync caught up with source: 2024-05-26T20:41:07.139 INFO:teuthology.orchestra.run.smithi137.stderr: local status: [, , , , , , , , 00000000003.212.7, , ] 2024-05-26T20:41:07.139 INFO:teuthology.orchestra.run.smithi137.stderr: remote markers: [, , , , , , , , 00000000003.212.7, , ] 2024-05-26T20:41:07.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:07.140+0000 7f1169698a80 0 bucket checkpoint complete 2024-05-26T20:41:07.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-29 zones={test-zone2, test-zone1} 2024-05-26T20:41:07.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:07.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:07.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:07.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-29 zones={test-zone2, test-zone1} 2024-05-26T20:41:07.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-29 zones={test-zone3, test-zone1} 2024-05-26T20:41:07.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:07.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:07.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:07.186 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-29 zones={test-zone3, test-zone1} 2024-05-26T20:41:07.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xmqxus-29 zones={test-zone3, test-zone2} 2024-05-26T20:41:07.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-05-26T20:41:07.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.193 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-05-26T20:41:07.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-05-26T20:41:07.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-05-26T20:41:07.248 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xmqxus-29 zones={test-zone3, test-zone2} 2024-05-26T20:41:07.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-05-26T20:41:07.248 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-30 2024-05-26T20:41:07.261 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-31 2024-05-26T20:41:07.285 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-32 2024-05-26T20:41:07.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-05-26T20:41:07.373 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-33 2024-05-26T20:41:07.386 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-34 2024-05-26T20:41:07.448 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-35 2024-05-26T20:41:07.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-05-26T20:41:07.545 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-36 2024-05-26T20:41:07.558 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-37 2024-05-26T20:41:07.620 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-38 2024-05-26T20:41:09.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-05-26T20:41:09.783 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-39 2024-05-26T20:41:09.801 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-40 2024-05-26T20:41:09.830 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-41 2024-05-26T20:41:09.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:09.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:09.857 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:09.932 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:09.932 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:09.995 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:41:10.006 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.846685_670.1", 2024-05-26T20:41:10.007 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:41:09.846685Z" 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:10.008 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:41:10.009 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:41:10.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:10.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:10.010 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:10.066 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:10.067 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:10.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:10.098+0000 7f06fa1c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:10.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:10.099+0000 7f06fa1c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:10.111 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:41:10.112 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:10.113 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:10.114 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:10.115 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:41:10Z" 2024-05-26T20:41:10.116 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:41:10.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:41:15.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:15.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:15.119 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:15.173 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:15.174 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:15.202 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:15.202+0000 7f0606798a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:15.203 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:15.203+0000 7f0606798a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:15.214 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:41:15.214 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:15.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.216 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:41:15.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:15.216 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:41:15.216 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:15.216 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:15.217 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:15.217 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:15.217 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:41:15.217 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:15.218 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:41:15.219 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:15.219 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:15.219 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:41:15.219 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:15.219 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:15.220 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:15.220 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:41:15.220 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:15.220 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:15.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:41:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:41:15.222 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:15.222 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:15.222 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:15.222 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:41:15.222 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:15.223 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:15.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:15.223 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:41:15.223 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:15.224 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:41:15.225 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:41:15.226 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:41:15.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:15.226 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:41:15Z" 2024-05-26T20:41:15.226 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:41:15.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:41:20.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:20.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:20.229 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:20.287 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:20.287 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:20.318 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:20.317+0000 7f5fa4868a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:20.321 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:41:20.318+0000 7f5fa4868a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.331 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:20.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:20.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756069.846685_670.1", 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:41:09.846685Z", 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:20.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:41:20.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:41:20Z" 2024-05-26T20:41:20.336 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:41:20.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:41:20.337 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:41:20.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:20.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:20.337 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:20.479+0000 7fbb01d0ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:20.480+0000 7fbb01d0ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:20.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:20.491 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:20.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:20.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:20.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:20.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:20.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:20Z" 2024-05-26T20:41:20.496 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:20.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:41:25.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:25.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:25.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:25.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:25.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:25.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:25.594+0000 7f24e006fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:25.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:25.595+0000 7f24e006fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:25.606 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:25.606 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:25.606 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:25.606 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:25.607 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:25.608 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:25.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:25.610 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:25Z" 2024-05-26T20:41:25.611 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:25.612 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:41:30.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:30.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:30.613 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:30.692+0000 7f0702430a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:30.693+0000 7f0702430a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:30.704 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:30.705 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:30.706 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:30.707 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.400971_630.1", 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.400971Z", 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:30.708 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:30.709 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:30.710 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:30.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:30.710 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:30Z" 2024-05-26T20:41:30.710 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:30.711 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.400971_630.1 2024-05-26T20:41:35.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:35.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:35.712 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:35.764 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:35.764 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:35.792+0000 7f574d8b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:35.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:35.793+0000 7f574d8b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.805 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:35.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:35.807 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.531167_638.1", 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.531167Z", 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:35.808 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:35.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:35.810 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:35Z" 2024-05-26T20:41:35.810 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:35.811 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.531167_638.1 2024-05-26T20:41:40.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:40.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:40.812 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:40.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:40.969+0000 7f9a9b581a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:40.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:40.970+0000 7f9a9b581a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:40.982 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:40.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:40.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:40.986 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:40.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:40.987 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:40Z" 2024-05-26T20:41:40.987 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:40.987 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:41:45.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:45.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:45.989 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:46.079+0000 7fdecb833a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:46.080+0000 7fdecb833a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:46.092 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:46Z" 2024-05-26T20:41:46.097 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:46.098 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.367915_621.1 2024-05-26T20:41:51.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:51.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:51.099 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:51.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:51.266+0000 7fc9e198ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:51.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:51.269+0000 7fc9e198ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:51.291 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:51.292 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:51.294 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:51.295 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:51.296 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:51.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:51.297 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:51Z" 2024-05-26T20:41:51.297 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:51.298 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:41:56.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:56.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:41:56.299 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:41:56.352 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:41:56.352 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:41:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:56.386+0000 7f36eca66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:41:56.387+0000 7f36eca66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:56.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:41:56.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:56.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:41:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:41:56Z" 2024-05-26T20:41:56.403 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:41:56.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:42:01.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:01.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:01.405 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:01.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:01.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:01.520+0000 7fb43d0e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:01.521+0000 7fb43d0e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:01.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:01.535 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:01.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:01.537 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:01.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:01.539 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:01.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:01.539 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:01Z" 2024-05-26T20:42:01.539 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:01.540 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:42:06.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:06.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:06.541 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:06.598 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:06.598 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:06.630+0000 7f7af548aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:06.631+0000 7f7af548aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:06.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:06.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:06.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:06.648 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:06Z" 2024-05-26T20:42:06.649 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:06.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:42:11.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:11.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:11.651 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:11.711 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:11.711 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:11.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:11.744+0000 7f2b228b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:11.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:11.745+0000 7f2b228b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:11.758 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:11.758 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:11.758 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:11.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:11.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:11.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:11Z" 2024-05-26T20:42:11.763 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:11.764 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:42:16.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:16.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:16.770 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:16.921+0000 7f3fd4277a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:16.922+0000 7f3fd4277a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:16.934 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:42:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:16Z" 2024-05-26T20:42:16.939 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:16.941 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296898_615.1 2024-05-26T20:42:21.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:21.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:21.942 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:21.996 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:21.996 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:22.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:22.027+0000 7f6a04248a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:22.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:22.028+0000 7f6a04248a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:22.040 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:22.040 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:22.040 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:22.040 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:22.041 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:22.042 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.385899_626.1", 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.385899Z", 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:22.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:22.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:22Z" 2024-05-26T20:42:22.045 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:22.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.385899_626.1 2024-05-26T20:42:27.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:27.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:27.047 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:27.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:27.142+0000 7f4584b56a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:27.144+0000 7f4584b56a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:27.156 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:27.156 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:27.156 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:27.156 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:27.157 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:27.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:27.159 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:42:27.160 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:27.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:27.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:27.164 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:27.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:27.164 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:27Z" 2024-05-26T20:42:27.164 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:27.165 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296898_615.1 2024-05-26T20:42:32.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:32.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:32.167 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:32.228 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:32.228 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:32.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:32.263+0000 7fc307bfea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:32.265+0000 7fc307bfea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:32.278 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:32Z" 2024-05-26T20:42:32.289 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:32.291 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:42:37.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:37.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:37.293 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:37.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:37.378+0000 7fef95540a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:37.379+0000 7fef95540a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:37.391 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:37.391 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:37.391 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:37.391 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:37.392 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:37.393 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:37.394 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:37.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:37.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:37.397 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:37Z" 2024-05-26T20:42:37.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:37.398 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.379895_623.1 2024-05-26T20:42:42.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:42.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:42.399 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:42.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:42.485+0000 7f9dc3016a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:42.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:42.487+0000 7f9dc3016a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:42Z" 2024-05-26T20:42:42.504 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:42.505 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:42:47.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:47.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:47.506 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:47.565 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:47.565 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:47.599+0000 7faa8b44aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:47.600+0000 7faa8b44aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:47.613 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:47.616 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:47Z" 2024-05-26T20:42:47.619 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:47.620 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.299030_616.1 2024-05-26T20:42:52.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:52.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:52.621 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:52.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:52.713+0000 7f7cd5d1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:52.714+0000 7f7cd5d1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:52.727 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:52.728 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:52.729 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:52.730 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:52.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:52.732 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:52.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:52.733 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:52Z" 2024-05-26T20:42:52.733 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:52.734 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.382628_624.1 2024-05-26T20:42:57.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:57.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:42:57.735 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:42:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:42:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:42:57.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:57.820+0000 7fa982096a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:57.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:42:57.821+0000 7fa982096a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:42:57.832 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.833 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:57.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:57.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.397876_628.1", 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.397876Z", 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:42:57.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:42:57.837 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:42:57.837 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:42:57.840 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:42:57.840 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:42:57.840 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:42:57.840 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:42:57Z" 2024-05-26T20:42:57.841 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:42:57.842 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.397876_628.1 2024-05-26T20:43:02.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:02.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:02.844 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:02.933+0000 7f16ceb90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:02.934+0000 7f16ceb90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:02.946 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:02.946 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:02.946 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:02.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:02.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:43:02.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:02.950 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:02Z" 2024-05-26T20:43:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:02.952 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:43:07.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:07.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:07.954 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:08.007 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:08.007 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:08.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:08.037+0000 7f213f625a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:08.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:08.038+0000 7f213f625a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:08Z" 2024-05-26T20:43:08.055 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:08.056 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:43:13.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:13.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:13.058 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:13.118 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:13.119 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:13.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:13.153+0000 7ff3e7d24a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:13.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:13.155+0000 7ff3e7d24a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:13.167 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:13.167 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:13.167 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:13.167 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:13.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:13.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:13.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:13.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.172 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:13.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:13.173 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:13.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:13.173 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:13Z" 2024-05-26T20:43:13.173 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:13.174 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:43:18.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:18.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:18.177 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:18.242 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:18.243 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:18.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:18.277+0000 7fd410456a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:18.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:18.278+0000 7fd410456a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:18.290 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:18.291 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:18.292 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:43:18.293 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:18Z" 2024-05-26T20:43:18.295 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:18.296 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.299030_616.1 2024-05-26T20:43:23.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:23.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:23.298 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:23.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:23.381+0000 7f8c372c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:23.382+0000 7f8c372c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:23.394 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:23.394 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:23.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:23.396 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:23.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:23.398 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:23.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:23.400 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:23.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:23.400 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:23Z" 2024-05-26T20:43:23.400 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:23.402 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296898_615.1 2024-05-26T20:43:28.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:28.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:28.403 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:28.460 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:28.460 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:28.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:28.493+0000 7ff212818a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:28.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:28.494+0000 7ff212818a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:28.506 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:28.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:28.509 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:28.511 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:28.511 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:28.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:28.511 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:28Z" 2024-05-26T20:43:28.511 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:28.512 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296898_615.1 2024-05-26T20:43:33.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:33.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:33.513 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:33.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:33.604+0000 7fa16bd49a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:33.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:33.605+0000 7fa16bd49a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:33.617 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:33.618 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:33.619 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:33.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:33.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:33.622 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:33.623 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:33.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:33.623 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:33Z" 2024-05-26T20:43:33.623 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:33.624 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.370262_622.1 2024-05-26T20:43:38.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:38.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:38.625 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:38.697 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:38.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:38.731+0000 7fee8667fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:38.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:38.732+0000 7fee8667fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:38.745 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:38.745 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:38.745 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:38.746 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:38.746 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:38.746 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:38.746 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:38.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:38.747 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:38.748 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:38.748 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:38.748 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:38.748 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:38.748 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:38.749 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:38.750 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:38.750 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:38.750 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:38.750 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:38.750 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:38.752 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T20:43:38.753 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:38.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:38.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:38.753 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:38.754 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:38.754 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:38.754 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:38.754 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:38.754 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:38.755 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:38.755 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:38.755 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:38.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:38.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:38.757 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:38Z" 2024-05-26T20:43:38.757 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:38.758 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.395458_627.1 2024-05-26T20:43:43.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:43.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:43.760 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:43.814 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:43.814 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:43.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:43.846+0000 7fbf4135ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:43.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:43.847+0000 7fbf4135ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:43Z" 2024-05-26T20:43:43.864 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:43.865 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:43:48.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:48.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:48.867 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:48.927 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:48.927 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:48.957+0000 7f32fa2dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:48.958+0000 7f32fa2dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:48.972 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:48.973 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:48.974 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:48.976 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:48.977 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:48.978 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:48.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:48.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:48.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:48.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:48.981 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:48.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.981 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:48.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:48.982 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:48.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:48.982 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:48Z" 2024-05-26T20:43:48.982 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:48.984 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:43:53.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:53.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:53.986 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:54.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:54.078+0000 7fe7c472ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:54.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:54.079+0000 7fe7c472ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:54.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.383646_625.1", 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.383646Z", 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:54Z" 2024-05-26T20:43:54.097 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:54.099 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.383646_625.1 2024-05-26T20:43:59.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:59.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:43:59.100 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:43:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:43:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:43:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:59.192+0000 7f1c95a76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:43:59.193+0000 7f1c95a76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:43:59.206 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:59.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:59.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:59.209 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:59.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:43:59.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:43:59Z" 2024-05-26T20:43:59.212 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:43:59.213 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:44:04.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:04.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:04.215 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:04.283 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:04.283 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:04.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:04.316+0000 7f531b4dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:04.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:04.317+0000 7f531b4dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:04.331 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:04.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:04.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:04.335 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:04Z" 2024-05-26T20:44:04.336 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:04.337 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296023_614.1 2024-05-26T20:44:09.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:09.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:09.338 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:09.423+0000 7f2781681a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:09.424+0000 7f2781681a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:09.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:09.438 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:09.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:09Z" 2024-05-26T20:44:09.442 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:09.443 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.457698_631.1 2024-05-26T20:44:14.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:14.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:14.445 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:14.537+0000 7f4e0bc61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:14.538+0000 7f4e0bc61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:14.551 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:14.551 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:14.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:44:14.554 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:14.555 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:14Z" 2024-05-26T20:44:14.556 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:14.557 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:44:19.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:19.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:19.559 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:19.617 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:19.618 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:19.651+0000 7efeec8c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:19.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:19.652+0000 7efeec8c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:19.665 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:19.665 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:19.665 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:19.665 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:19.665 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:19.666 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:19.667 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:19.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:19.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:19Z" 2024-05-26T20:44:19.670 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:19.671 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.276118_612.1 2024-05-26T20:44:24.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:24.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:24.672 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:24.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:24.766+0000 7f93d7079a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:24.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:24.767+0000 7f93d7079a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:24.779 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:24.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:24.781 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:24.782 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:24.782 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:24.782 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:24.782 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:24.782 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:24.783 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:24.784 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:24.784 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:24.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:24.784 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:24.784 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:24.785 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:24.785 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:24.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:24.786 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:24.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:24Z" 2024-05-26T20:44:24.790 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:24.792 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:44:29.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:29.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:29.794 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:29.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:29.973+0000 7f9717382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:29.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:29.973+0000 7f9717382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:29.986 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:29.986 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:29.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:29.988 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:29.990 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:29.991 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:29.992 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:29.994 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:29.995 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:29.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:29.996 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:29Z" 2024-05-26T20:44:29.996 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:29.997 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:44:34.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:34.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:34.999 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:35.087+0000 7f1c4a264a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:35.088+0000 7f1c4a264a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:35Z" 2024-05-26T20:44:35.111 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:35.111 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:44:40.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:40.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:40.113 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:40.207+0000 7ff10a20fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:40.209+0000 7ff10a20fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:40.223 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:40.224 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.531167_638.1", 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.531167Z", 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:40.225 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:40.226 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:40.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:40.227 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:40Z" 2024-05-26T20:44:40.227 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:40.228 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.531167_638.1 2024-05-26T20:44:45.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:45.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:45.229 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:45.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:45.320+0000 7f4122106a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:45.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:45.321+0000 7f4122106a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:45.333 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:45Z" 2024-05-26T20:44:45.339 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:45.340 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296023_614.1 2024-05-26T20:44:50.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:50.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:50.341 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:50.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:50.447+0000 7f7eb684ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:50.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:50.448+0000 7f7eb684ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:50.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:50.464 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:50.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:50.466 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:50.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:50.468 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:50Z" 2024-05-26T20:44:50.468 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:50.469 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.367915_621.1 2024-05-26T20:44:55.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:55.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:44:55.470 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:44:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:44:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:44:55.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:55.598+0000 7fb05897da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:44:55.599+0000 7fb05897da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:44:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:55.615 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:44:55.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:44:55.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:44:55.618 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:44:55.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:44:55.619 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:44:55Z" 2024-05-26T20:44:55.619 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:44:55.620 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.382628_624.1 2024-05-26T20:45:00.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:00.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:00.622 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:00.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:00.704+0000 7f0480a7aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:00.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:00.705+0000 7f0480a7aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:00.717 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:00.717 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:00.717 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:00.718 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:00.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:00.720 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:00.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:00Z" 2024-05-26T20:45:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:00.723 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:45:05.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:05.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:05.725 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:05.783 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:05.783 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:05.816+0000 7f00b61b2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:05.817+0000 7f00b61b2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:05.829 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:05.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:05.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:05.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:05.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:05Z" 2024-05-26T20:45:05.834 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:05.835 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:45:10.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:10.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:10.836 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:10.895 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:10.929+0000 7f71d4203a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:10.930+0000 7f71d4203a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.943 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:10.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:10.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.947 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:10.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:10.948 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:10.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:10.948 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:10Z" 2024-05-26T20:45:10.948 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:10.949 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.309042_617.1 2024-05-26T20:45:15.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:15.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:15.951 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:16.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:16.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:16.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:16.043+0000 7fafbf4dca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:16.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:16.044+0000 7fafbf4dca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:16.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:16.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:16.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:16.060 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:16.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:16Z" 2024-05-26T20:45:16.062 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:16.063 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.362486_620.1 2024-05-26T20:45:21.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:21.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:21.066 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:21.127 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:21.161+0000 7f6d71685a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:21.162+0000 7f6d71685a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:21.175 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:45:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:21Z" 2024-05-26T20:45:21.180 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:21.182 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.299030_616.1 2024-05-26T20:45:26.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:26.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:26.183 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:26.295 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:26.296 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:26.363+0000 7f71c6a08a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:26.364+0000 7f71c6a08a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:26.376 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:26.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:26.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:26.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:26.380 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:26.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:26.382 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:26Z" 2024-05-26T20:45:26.382 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:26.383 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:45:31.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:31.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:31.384 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:31.473+0000 7f69d86c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:31.474+0000 7f69d86c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:31.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:31.489 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.531167_638.1", 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.531167Z", 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:31.490 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:31Z" 2024-05-26T20:45:31.492 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:31.493 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.531167_638.1 2024-05-26T20:45:36.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:36.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:36.494 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:36.586+0000 7f88632fea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:36.587+0000 7f88632fea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:36.600 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:36.600 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:36Z" 2024-05-26T20:45:36.606 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:36.607 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.299030_616.1 2024-05-26T20:45:41.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:41.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:41.608 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:41.701+0000 7f576430ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:41.702+0000 7f576430ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:41Z" 2024-05-26T20:45:41.720 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:41.721 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296023_614.1 2024-05-26T20:45:46.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:46.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:46.723 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:46.814+0000 7f182179da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:46.815+0000 7f182179da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:46.828 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:46.828 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:46.828 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:46.829 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:46.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:46.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:46.832 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:46Z" 2024-05-26T20:45:46.833 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:46.834 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:45:51.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:51.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:51.835 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:51.981+0000 7fe423a07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:51.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:51.982+0000 7fe423a07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:51.995 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:51.995 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:51.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:51.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:51.998 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:51.999 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:51Z" 2024-05-26T20:45:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:52.003 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.293638_613.1 2024-05-26T20:45:57.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:57.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:45:57.005 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:45:57.058 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:45:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:45:57.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:57.090+0000 7f0d25734a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:57.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:45:57.092+0000 7f0d25734a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:45:57.104 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:45:57.104 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:45:57.104 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:45:57.105 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:57.106 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:57.107 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:45:57.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:45:57Z" 2024-05-26T20:45:57.109 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:45:57.110 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.296898_615.1 2024-05-26T20:46:02.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:02.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:02.112 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:02.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:02.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:02.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:02.201+0000 7f2a1437da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:02.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:02.202+0000 7f2a1437da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.214 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:02.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:02.216 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:02.217 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:02.218 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:02.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:02.219 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:02Z" 2024-05-26T20:46:02.219 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:02.220 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.299030_616.1 2024-05-26T20:46:07.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:07.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:07.222 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:07.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:07.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:07.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:07.311+0000 7fd263a67a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:07.312+0000 7fd263a67a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.325 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:07.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:07.327 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.542149_640.1", 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.542149Z", 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:07.328 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:07Z" 2024-05-26T20:46:07.330 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:07.331 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.542149_640.1 2024-05-26T20:46:12.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:12.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:12.333 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:12.391 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:12.391 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:12.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:12.424+0000 7f8b2ee5aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:12.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:12.425+0000 7f8b2ee5aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:12.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:12.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:12.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:12.441 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:12.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:12.443 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:12.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:12.443 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:12Z" 2024-05-26T20:46:12.443 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:12.444 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:46:17.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:17.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:17.445 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:17.503 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:17.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:17.537+0000 7f91284aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:17.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:17.538+0000 7f91284aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:17.551 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:17.555 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:17Z" 2024-05-26T20:46:17.556 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:17.557 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.360000_619.1 2024-05-26T20:46:22.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:22.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:22.558 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:22.626 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:22.626 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:22.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:22.661+0000 7f882fad3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:22.664+0000 7f882fad3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:22.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:22.678 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:22.679 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:22.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:22.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:22.682 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:22Z" 2024-05-26T20:46:22.682 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:22.683 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756069.846685_670.1 target=1_1716756067.457698_631.1 2024-05-26T20:46:27.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-05-26T20:46:27.685 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-42 2024-05-26T20:46:27.731 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-43 2024-05-26T20:46:27.759 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-44 2024-05-26T20:46:27.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:27.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:27.826 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:27.880 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:27.880 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:46:28.017 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:46:28.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:28.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.775409_728.1", 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:46:27.775409Z" 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:28.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:28.019 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:28.019 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:46:28.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:28.019 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:46:28.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:46:28.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:28.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:28.020 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:28.077 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:28.077 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:28.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:28.109+0000 7fb6ce5cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:28.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:28.110+0000 7fb6ce5cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:46:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:46:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:46:28Z" 2024-05-26T20:46:28.128 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:46:28.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756387.702842_717.1 2024-05-26T20:46:33.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:33.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:33.130 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:33.191 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:33.191 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:33.227 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:33.227+0000 7f98c60b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:33.228 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:33.228+0000 7f98c60b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:33.241 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:33.242 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:46:33.243 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:33.245 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:46:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:33.247 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:46:33Z" 2024-05-26T20:46:33.247 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:46:33.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756387.702842_717.1 2024-05-26T20:46:38.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:38.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:38.249 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:38.307 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:38.307 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:38.338 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:38.339+0000 7f22385ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:38.339 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:46:38.340+0000 7f22385ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756387.775409_728.1", 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:46:27.775409Z", 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:46:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:46:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:46:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:46:38Z" 2024-05-26T20:46:38.358 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:46:38.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:46:38.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:46:38.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:38.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:38.363 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:38.414 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:38.414 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:38.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:38.525+0000 7f85e36a2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:38.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:38.526+0000 7f85e36a2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:38.543 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:46:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:38Z" 2024-05-26T20:46:38.553 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:38.554 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:46:43.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:43.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:43.556 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:43.608 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:43.608 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:43.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:43.637+0000 7f53f7329a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:43.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:43.638+0000 7f53f7329a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:43.650 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:43.650 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:43.651 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:43Z" 2024-05-26T20:46:43.657 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:43.658 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:46:48.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:48.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:48.660 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:48.815+0000 7f2ca1b25a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:48.816+0000 7f2ca1b25a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:48.830 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:48.830 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:48.830 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:48.830 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:48.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:48.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:48.834 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:48Z" 2024-05-26T20:46:48.835 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:48.836 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.299030_616.1 2024-05-26T20:46:53.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:53.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:53.838 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:53.918+0000 7fa832f76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:53.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:53.919+0000 7fa832f76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:53.931 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:53.931 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:53.931 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:53.931 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:53.932 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:53.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.398810_629.1", 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:53.934 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.398810Z", 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:53.935 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:53Z" 2024-05-26T20:46:53.936 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:53.937 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.398810_629.1 2024-05-26T20:46:58.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:58.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:46:58.939 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:46:58.993 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:46:58.993 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:46:59.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:59.024+0000 7fc21bbd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:59.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:46:59.025+0000 7fc21bbd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:46:59.037 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:46:59.037 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:46:59.037 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:46:59.037 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:59.038 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:46:59.039 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:59.040 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:46:59.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:46:59.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:46:59.043 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:46:59Z" 2024-05-26T20:46:59.043 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:46:59.044 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296023_614.1 2024-05-26T20:47:04.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:04.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:04.045 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:04.203+0000 7efe9324da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:04.204+0000 7efe9324da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:47:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:04Z" 2024-05-26T20:47:04.223 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:04.224 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296023_614.1 2024-05-26T20:47:09.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:09.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:09.226 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:09.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:09.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:09.324+0000 7f5fd70d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:09.335+0000 7f5fd70d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:09.413 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:09.413 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:09.414 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:09.415 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:09.416 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:09.417 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:09Z" 2024-05-26T20:47:09.418 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:09.419 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.370262_622.1 2024-05-26T20:47:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:14.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:14.421 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:14.502+0000 7f3c2bacba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:14.503+0000 7f3c2bacba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:14.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:14.518 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:14.519 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:14.520 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:14.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:14.521 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:14Z" 2024-05-26T20:47:14.521 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:14.522 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296023_614.1 2024-05-26T20:47:19.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:19.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:19.523 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:19.577 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:19.577 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:19.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:19.608+0000 7f11b05a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:19.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:19.609+0000 7f11b05a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:19.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:19.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:47:19.625 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:19.626 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:19Z" 2024-05-26T20:47:19.627 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:19.628 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:47:24.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:24.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:24.630 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:24.717+0000 7f6518f8ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:24.718+0000 7f6518f8ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:24.731 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:24.732 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:24.732 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:24.732 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:24.732 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:24.732 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:24.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:24.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.733 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:24.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:24.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:24.735 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:24.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:47:24.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:24.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:24.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:24.741 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:24Z" 2024-05-26T20:47:24.741 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:24.742 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.299030_616.1 2024-05-26T20:47:29.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:29.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:29.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:29.837+0000 7f0bfdf1da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:29.838+0000 7f0bfdf1da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:29.852 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:29.853 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:29.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:47:29.855 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:29.856 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:29.857 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:29.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:29.858 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:29Z" 2024-05-26T20:47:29.858 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:29.859 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.311444_618.1 2024-05-26T20:47:34.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:34.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:34.861 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:34.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:34.957+0000 7f00e0eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:34.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:34.965+0000 7f00e0eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:35.054 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:35.054 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:35.054 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:35.055 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:35.056 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:35.057 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:34Z" 2024-05-26T20:47:35.060 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:35.060 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.309042_617.1 2024-05-26T20:47:40.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:40.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:40.062 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:40.120 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:40.120 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:40.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:40.153+0000 7fb80e427a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:40.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:40.154+0000 7fb80e427a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:40.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:40.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:40.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:40.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:40.172 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:40Z" 2024-05-26T20:47:40.172 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:40.173 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.309042_617.1 2024-05-26T20:47:45.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:45.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:45.175 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:45.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:45.257+0000 7f6abb99da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:45.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:45.258+0000 7f6abb99da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:45Z" 2024-05-26T20:47:45.276 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:45.276 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:47:50.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:50.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:50.278 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:50.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:50.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:50.359+0000 7fc1d4e51a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:50.360+0000 7fc1d4e51a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:50.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:50.374 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:50.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.474724_635.1", 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.474724Z", 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:50.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:50Z" 2024-05-26T20:47:50.378 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:50.379 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.474724_635.1 2024-05-26T20:47:55.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:55.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:47:55.381 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:47:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:47:55.437 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:47:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:55.470+0000 7fd0b5cb9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:47:55.472+0000 7fd0b5cb9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:47:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:47:55.489 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:47:55.490 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:47:55.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:47:55.490 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:47:55Z" 2024-05-26T20:47:55.490 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:47:55.491 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296898_615.1 2024-05-26T20:48:00.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:00.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:00.492 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:00.550 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:00.550 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:00.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:00.583+0000 7f6bfa1cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:00.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:00.584+0000 7f6bfa1cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:00.598 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:00.599 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:00.600 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:00.601 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:00.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:00.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:00.604 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:00.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:00.604 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:00Z" 2024-05-26T20:48:00.604 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:00.605 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.311444_618.1 2024-05-26T20:48:05.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:05.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:05.607 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:05.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:05.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:05.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:05.690+0000 7fb8b795ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:05.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:05.691+0000 7fb8b795ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:05.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:05.707 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:05.708 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:05Z" 2024-05-26T20:48:05.709 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:05.709 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.362486_620.1 2024-05-26T20:48:10.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:10.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:10.711 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:10.805+0000 7f1243eb4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:10.806+0000 7f1243eb4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:10.820 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:10Z" 2024-05-26T20:48:10.825 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:10.826 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.311444_618.1 2024-05-26T20:48:15.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:15.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:15.827 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:15.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:15.908+0000 7fbd5c2d1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:15.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:15.909+0000 7fbd5c2d1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:15.922 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:15.923 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:15.924 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:48:15.925 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:15.926 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:15Z" 2024-05-26T20:48:15.927 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:15.928 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.379895_623.1 2024-05-26T20:48:20.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:20.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:20.930 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:21.022+0000 7f85d2b74a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:21.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:21.023+0000 7f85d2b74a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:21.037 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:21.037 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:21.037 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:21.038 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:21.039 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.551890_641.1", 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.551890Z", 2024-05-26T20:48:21.040 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:21.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:21Z" 2024-05-26T20:48:21.042 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:21.043 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.551890_641.1 2024-05-26T20:48:26.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:26.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:26.045 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:26.129 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:26.129 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:26.214+0000 7fa827399a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:26.215+0000 7fa827399a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:26.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:26.239 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:26.240 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:26.241 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:26.242 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:26.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:26.243 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:26Z" 2024-05-26T20:48:26.243 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:26.243 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.309042_617.1 2024-05-26T20:48:31.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:31.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:31.244 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:31.297 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:31.297 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:31.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:31.327+0000 7fa98401ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:31.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:31.328+0000 7fa98401ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:31.340 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:31.340 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:31.340 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:31.340 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:31.341 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:31.342 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:31.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:31.345 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:31Z" 2024-05-26T20:48:31.346 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:31.347 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.370262_622.1 2024-05-26T20:48:36.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:36.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:36.349 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:36.403 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:36.403 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:36.433+0000 7f9289e1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:36.434+0000 7f9289e1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:36Z" 2024-05-26T20:48:36.452 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:36.453 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.367915_621.1 2024-05-26T20:48:41.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:41.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:41.454 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:41.507 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:41.507 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:41.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:41.538+0000 7ff2f949ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:41.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:41.539+0000 7ff2f949ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:41.552 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:41.552 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:41.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:41.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:41.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:41.556 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:41.557 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:41Z" 2024-05-26T20:48:41.558 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:41.559 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.360000_619.1 2024-05-26T20:48:46.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:46.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:46.560 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:46.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:46.654+0000 7f1ae942ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:46.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:46.655+0000 7f1ae942ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:46Z" 2024-05-26T20:48:46.674 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:46.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296898_615.1 2024-05-26T20:48:51.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:51.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:51.676 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:51.794+0000 7f2d1e625a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:51.795+0000 7f2d1e625a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:51.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:51.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:51.811 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:51.812 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:51.813 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:51.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:51.813 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:51Z" 2024-05-26T20:48:51.813 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:51.814 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.367915_621.1 2024-05-26T20:48:56.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:56.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:48:56.815 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:48:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:48:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:48:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:56.909+0000 7fbd37d03a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:48:56.910+0000 7fbd37d03a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:48:56.924 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:48:56.924 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:48:56.924 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:48:56.924 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:56.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:56.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:56.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:48:56.928 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:48:56Z" 2024-05-26T20:48:56.929 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:48:56.930 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.360000_619.1 2024-05-26T20:49:01.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:01.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:01.932 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:02.026+0000 7f19c3a08a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:02.027+0000 7f19c3a08a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:02.041 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:02.042 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:02.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:49:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:02.045 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:02Z" 2024-05-26T20:49:02.046 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:02.047 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.311444_618.1 2024-05-26T20:49:07.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:07.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:07.049 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:07.147+0000 7fa71747ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:07.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:07.149+0000 7fa71747ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:07.162 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:07.162 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:07.162 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:07.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:49:07.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:07.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:07Z" 2024-05-26T20:49:07.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:07.168 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296023_614.1 2024-05-26T20:49:12.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:12.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:12.170 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:12.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:12.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:12.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:12.265+0000 7f7c7c9a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:12.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:12.266+0000 7f7c7c9a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:12.281 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:12.281 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:12.281 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:12.281 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:12.282 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:12.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:12.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:12.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:12Z" 2024-05-26T20:49:12.286 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:12.287 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.311444_618.1 2024-05-26T20:49:17.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:17.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:17.288 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:17.373+0000 7f10b7e4ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:17.374+0000 7f10b7e4ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:17.388 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:17.389 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:17.390 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:17.391 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:17Z" 2024-05-26T20:49:17.392 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:17.393 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.457698_631.1 2024-05-26T20:49:22.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:22.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:22.395 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:22.447 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:22.448 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:22.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:22.479+0000 7f947f9b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:22.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:22.480+0000 7f947f9b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:22.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:22Z" 2024-05-26T20:49:22.498 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:22.499 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.360000_619.1 2024-05-26T20:49:27.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:27.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:27.500 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:27.594+0000 7f8480266a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:27.595+0000 7f8480266a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:27.619 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:27.619 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:27.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:27.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.474724_635.1", 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:27.622 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.474724Z", 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:27.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:27Z" 2024-05-26T20:49:27.624 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:27.625 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.474724_635.1 2024-05-26T20:49:32.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:32.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:32.627 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:32.720+0000 7f845d68fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:32.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:32.721+0000 7f845d68fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:32.735 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:32.735 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:32.735 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:32.735 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:32.735 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:32.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:32.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:49:32.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:32.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:32Z" 2024-05-26T20:49:32.740 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:32.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:49:37.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:37.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:37.743 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:37.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:37.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:37.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:37.855+0000 7f6d38882a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:37.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:37.856+0000 7f6d38882a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:37.869 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:37.869 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:37.869 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:37.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:49:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:37.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:37Z" 2024-05-26T20:49:37.874 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:37.875 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:49:42.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:42.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:42.877 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:42.928 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:42.928 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:42.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:42.959+0000 7ff07590da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:42.960+0000 7ff07590da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:42.972 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:42Z" 2024-05-26T20:49:42.978 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:42.979 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.360000_619.1 2024-05-26T20:49:47.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:47.980 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:48.039 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:48.039 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:48.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:48.074+0000 7f217e389a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:48.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:48.075+0000 7f217e389a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:48.089 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:48.089 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:48.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.091 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:48.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:48.091 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:48.091 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:48.091 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:48.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:48.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:48.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:48.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:49:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:48.098 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:48.098 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:48.098 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:48.098 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:48.099 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:48.100 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:48.100 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:48.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.100 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:48.100 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:48.101 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:48.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:48.101 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:48Z" 2024-05-26T20:49:48.101 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:48.103 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:49:53.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:53.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:53.104 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:53.163 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:53.164 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:53.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:53.198+0000 7f01c1195a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:53.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:53.199+0000 7f01c1195a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:53.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:53.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:53Z" 2024-05-26T20:49:53.218 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:53.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.309042_617.1 2024-05-26T20:49:58.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:58.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:49:58.221 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:49:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:49:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:49:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:58.316+0000 7fef244aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:58.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:49:58.316+0000 7fef244aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:49:58.330 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:58.332 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:49:58.333 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:49:58.334 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:49:58.335 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:49:58Z" 2024-05-26T20:49:58.336 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:49:58.337 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:50:03.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:03.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:03.338 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:03.437 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:03.481+0000 7f8a027b2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:03.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:03.482+0000 7f8a027b2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:03.495 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:03.495 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:03.495 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:03.495 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:03.495 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:03.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:03.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:03.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:03.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:03Z" 2024-05-26T20:50:03.500 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:03.501 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756068.652631_655.1 2024-05-26T20:50:08.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:08.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:08.502 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:08.554 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:08.554 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:08.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:08.586+0000 7f9f89721a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:08.587+0000 7f9f89721a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:08.600 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:08.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:08.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:08.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:08Z" 2024-05-26T20:50:08.604 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:08.605 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.367915_621.1 2024-05-26T20:50:13.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:13.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:13.607 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:13.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:13.703+0000 7f976d55ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:13.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:13.704+0000 7f976d55ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:13.718 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:13.718 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:13.718 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:13.719 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:13.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:13.721 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:13.722 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:13Z" 2024-05-26T20:50:13.723 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:13.724 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:50:18.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:18.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:18.725 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:18.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:18.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:18.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:18.820+0000 7f6c41879a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:18.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:18.821+0000 7f6c41879a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:18.835 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:18.835 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:18.836 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:18.837 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:18.838 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:18.839 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:18Z" 2024-05-26T20:50:18.840 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:18.841 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.629964_649.1 2024-05-26T20:50:23.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:23.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:23.843 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:23.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:23.936+0000 7fdb3c220a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:23.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:23.938+0000 7fdb3c220a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:23.952 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:23.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:23.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:23.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:23.956 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:23Z" 2024-05-26T20:50:23.957 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:23.958 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.382628_624.1 2024-05-26T20:50:28.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:28.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:28.959 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:29.012 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:29.012 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:29.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:29.043+0000 7f5b7a44da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:29.044+0000 7f5b7a44da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:29.057 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:29.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:29.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:29Z" 2024-05-26T20:50:29.062 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:29.063 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.367915_621.1 2024-05-26T20:50:34.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:34.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:34.064 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:34.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:34.158+0000 7ff26f75ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:34.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:34.159+0000 7ff26f75ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:34.173 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:34.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:34.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:34.178 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:34Z" 2024-05-26T20:50:34.179 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:34.180 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.299030_616.1 2024-05-26T20:50:39.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:39.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:39.181 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:39.237 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:39.238 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:39.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:39.270+0000 7fa89d7d8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:39.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:39.271+0000 7fa89d7d8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:39.285 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:39.285 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:39.285 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:39.286 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:39.287 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.460055_632.1", 2024-05-26T20:50:39.288 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.460055Z", 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:39.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:39.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:39.291 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:39.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:39.291 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:39Z" 2024-05-26T20:50:39.291 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:39.292 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.460055_632.1 2024-05-26T20:50:44.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:44.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:44.295 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:44.376 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:44.377 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:44.413+0000 7efd65fcba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:44.414+0000 7efd65fcba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:44.427 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:44Z" 2024-05-26T20:50:44.433 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:44.433 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.362486_620.1 2024-05-26T20:50:49.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:49.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:49.435 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:49.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:49.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:49.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:49.523+0000 7f44b2d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:49.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:49.524+0000 7f44b2d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:49.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:49.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:50:49.540 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:49.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:49Z" 2024-05-26T20:50:49.542 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:49.543 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.309042_617.1 2024-05-26T20:50:54.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:54.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:54.545 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:54.603 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:54.603 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:54.638+0000 7f257efaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:54.639+0000 7f257efaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:54.653 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:54.653 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:54.653 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:54.653 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:54.653 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:54.654 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:54.655 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:54.656 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:54.657 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:54Z" 2024-05-26T20:50:54.658 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:54.659 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:50:59.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:59.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:50:59.662 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:50:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:50:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:50:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:59.755+0000 7ff973f67a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:50:59.756+0000 7ff973f67a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:50:59.770 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:50:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:59.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:50:59.773 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:59.774 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:50:59.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:50:59.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:50:59.777 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:50:59Z" 2024-05-26T20:50:59.777 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:50:59.778 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.367915_621.1 2024-05-26T20:51:04.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:04.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:04.779 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:04.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:04.970+0000 7fb3085c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:04.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:04.973+0000 7fb3085c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:04.997 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:04.998 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:04.999 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:51:05.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:05.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:04Z" 2024-05-26T20:51:05.002 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:05.004 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:51:10.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:10.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:10.005 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:10.057 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:10.057 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:10.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:10.093+0000 7f717a887a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:10.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:10.094+0000 7f717a887a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:10.106 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:10.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:10.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:10.109 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:10.110 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:10.111 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:10.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:10.112 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:10Z" 2024-05-26T20:51:10.112 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:10.113 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.296898_615.1 2024-05-26T20:51:15.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:15.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:15.114 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:15.175 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:15.175 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:15.206+0000 7f13c316da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:15.207+0000 7f13c316da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:15.220 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:15.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:15.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:15.223 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:15.224 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:15.225 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:15Z" 2024-05-26T20:51:15.226 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:15.227 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:51:20.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:20.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:20.229 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:20.349+0000 7f8da595aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:20.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:20.350+0000 7f8da595aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:20Z" 2024-05-26T20:51:20.387 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:20.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:51:25.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:25.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:25.390 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:25.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:25.484+0000 7f2202772a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:25.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:25.485+0000 7f2202772a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:25.499 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:25.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:25.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:25.503 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:25.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:25.504 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:25Z" 2024-05-26T20:51:25.504 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:25.505 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.276118_612.1 2024-05-26T20:51:30.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:30.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:30.506 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:30.566 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:30.566 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:30.600+0000 7f9704596a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:30.601+0000 7f9704596a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:30.615 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:30.615 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:30.615 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:30.616 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:30.617 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:30.618 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:30.619 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:30Z" 2024-05-26T20:51:30.620 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:30.621 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.362486_620.1 2024-05-26T20:51:35.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:35.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:35.623 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:35.689 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:35.689 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:35.723+0000 7fb32c38fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:35.724+0000 7fb32c38fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:35.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:35.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.400971_630.1", 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.400971Z", 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:35.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:35.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:35.742 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:35Z" 2024-05-26T20:51:35.742 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:35.743 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.400971_630.1 2024-05-26T20:51:40.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:40.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:40.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:40.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:40.837+0000 7f63be7e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:40.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:40.838+0000 7f63be7e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:40.852 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:40.852 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:40.852 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:40.852 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:40.854 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:40.855 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:40Z" 2024-05-26T20:51:40.857 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:40.858 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756387.775409_728.1 target=1_1716756067.293638_613.1 2024-05-26T20:51:45.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2024-05-26T20:51:45.861 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-45 2024-05-26T20:51:45.908 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-46 2024-05-26T20:51:45.946 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-47 2024-05-26T20:51:46.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:46.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:46.016 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:46.101 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:51:46.127 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:51:46.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:51:45.971680Z" 2024-05-26T20:51:46.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:46.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:46.129 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:46.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:51:46.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:46.129 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:51:46.129 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:51:46.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:46.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:46.130 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:46.188 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:46.189 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:46.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:51:46.218+0000 7ff6aeefba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:46.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:51:46.219+0000 7ff6aeefba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:46.231 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:51:46.231 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:51:46.231 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:51:46.231 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:51:46.231 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:46.232 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:46.233 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.895718_762.1", 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:46.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.895718Z", 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:46.235 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:51:46.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:51:46.237 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:51:46.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:46.237 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:51:46Z" 2024-05-26T20:51:46.237 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:51:46.238 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T20:51:46.232+0000 7f25336ee700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T20:51:46.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756705.895718_762.1 2024-05-26T20:51:51.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:51.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:51.240 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:51.329 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:51:51.329+0000 7f147bdefa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:51.330 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:51:51.330+0000 7f147bdefa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:51:51.343 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:51:51.344 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:51.345 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:51:51.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:51:51.347 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:51:51Z" 2024-05-26T20:51:51.348 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:51:51.348 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:51:51.349 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:51:51.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:51.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:51.349 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:51.436+0000 7f6339a92a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:51.437+0000 7f6339a92a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:51.450 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:51.450 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:51.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:51.454 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:51Z" 2024-05-26T20:51:51.455 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:51.456 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.379895_623.1 2024-05-26T20:51:56.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:56.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:51:56.458 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:51:56.511 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:51:56.512 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:51:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:56.544+0000 7f3d9277ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:56.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:51:56.545+0000 7f3d9277ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:51:56.558 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:56.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:56.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.400971_630.1", 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:56.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.400971Z", 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:51:56.562 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:51:56Z" 2024-05-26T20:51:56.563 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:51:56.564 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.400971_630.1 2024-05-26T20:52:01.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:01.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:01.566 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:01.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:01.652+0000 7f88bc110a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:01.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:01.653+0000 7f88bc110a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:01.666 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:01.667 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:01.668 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:01.669 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:01.670 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:01Z" 2024-05-26T20:52:01.671 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:01.672 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.457698_631.1 2024-05-26T20:52:06.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:06.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:06.673 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:06.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:06.849+0000 7f1150b45a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:06.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:06.851+0000 7f1150b45a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:06.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:06.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:06Z" 2024-05-26T20:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:06.872 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:52:11.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:11.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:11.873 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:11.958+0000 7fa92ad58a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:11.959+0000 7fa92ad58a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.972 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:11.974 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.383646_625.1", 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:11.975 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.383646Z", 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:11.976 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:11.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:11.978 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:11.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:11.978 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:11Z" 2024-05-26T20:52:11.978 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:11.979 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.383646_625.1 2024-05-26T20:52:16.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:16.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:16.981 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:17.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:17.075+0000 7f34a5d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:17.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:17.076+0000 7f34a5d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:17.090 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:17.091 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:17.092 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:17.094 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:17Z" 2024-05-26T20:52:17.095 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:17.096 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:52:22.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:22.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:22.098 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:22.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:22.191+0000 7f14da18da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:22.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:22.192+0000 7f14da18da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:22.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:22.209 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:22.210 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.211 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:22Z" 2024-05-26T20:52:22.213 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:22.215 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.395458_627.1 2024-05-26T20:52:27.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:27.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:27.216 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:27.360+0000 7f95ed38da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:27.363+0000 7f95ed38da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:27.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:27.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:27.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:27.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:27.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:27.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:27.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:27Z" 2024-05-26T20:52:27.429 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:27.430 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:52:32.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:32.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:32.431 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:32.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:32.514+0000 7f3018f9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:32.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:32.515+0000 7f3018f9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:32.528 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:32.528 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:32.528 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:32.529 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:32.530 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:32.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:32.533 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:32.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:32.534 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:32Z" 2024-05-26T20:52:32.534 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:32.535 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.367915_621.1 2024-05-26T20:52:37.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:37.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:37.536 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:37.626+0000 7f0f7f23ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:37.627+0000 7f0f7f23ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:37.641 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:37.642 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:37.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:37.644 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:37.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:37Z" 2024-05-26T20:52:37.647 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:37.648 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.296898_615.1 2024-05-26T20:52:42.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:42.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:42.650 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:42.707 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:42.707 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:42.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:42.741+0000 7fdb0939ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:42.742+0000 7fdb0939ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:42.756 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:42Z" 2024-05-26T20:52:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:42.762 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.379895_623.1 2024-05-26T20:52:47.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:47.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:47.764 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:47.819 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:47.819 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:47.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:47.853+0000 7f2be3f95a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:47.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:47.854+0000 7f2be3f95a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:47.868 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:47.868 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:47.868 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:47.869 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:47.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:47.871 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:47.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:47Z" 2024-05-26T20:52:47.873 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:47.874 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:52:52.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:52.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:52.875 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:53.049+0000 7f5eb9c82a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:53.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:53.050+0000 7f5eb9c82a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:53.064 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:53.065 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:53.067 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:53.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:53Z" 2024-05-26T20:52:53.069 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:53.070 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.379895_623.1 2024-05-26T20:52:58.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:58.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:52:58.072 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:52:58.124 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:52:58.124 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:52:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:58.155+0000 7fde8dcb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:52:58.157+0000 7fde8dcb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:52:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:52:58.174 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:52:58.174 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:52:58.174 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:52:58.174 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:52:58.175 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:52:58.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:58.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.175 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:52:58.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:58.176 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:52:58.177 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:58.177 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:58.177 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:52:58.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:58.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:52:58.178 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:58.178 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:52:58.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:58.179 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:52:58.180 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:58.181 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:58.182 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T20:52:58.182 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:58.182 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:58.182 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:58.182 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:52:58.183 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:52:58.184 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:52:58.184 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:58.185 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:52:58.186 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:52:58Z" 2024-05-26T20:52:58.187 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:52:58.187 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.395458_627.1 2024-05-26T20:53:03.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:03.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:03.189 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:03.240 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:03.240 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:03.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:03.272+0000 7fccaba97a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:03.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:03.273+0000 7fccaba97a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:03.286 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:03.286 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:03.286 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:03.286 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:03.287 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:03.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:03.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:03.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:03Z" 2024-05-26T20:53:03.291 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:03.292 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:53:08.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:08.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:08.294 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:08.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:08.377+0000 7f60bc643a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:08.378+0000 7f60bc643a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:08.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:08.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:08.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.396 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:08.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:08.397 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:08.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:08.397 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:08Z" 2024-05-26T20:53:08.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:08.398 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:53:13.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:13.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:13.399 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:13.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:13.492+0000 7efc8eb79a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:13.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:13.493+0000 7efc8eb79a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:13.507 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:13.508 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:13.509 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:13.510 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:13.511 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:13Z" 2024-05-26T20:53:13.512 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:13.513 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:53:18.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:18.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:18.515 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:18.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:18.611+0000 7effcbd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:18.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:18.612+0000 7effcbd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:18.627 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:18.628 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:18.629 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:18.630 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:18Z" 2024-05-26T20:53:18.631 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:18.632 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.299030_616.1 2024-05-26T20:53:23.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:23.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:23.634 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:23.696 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:23.697 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:23.731+0000 7fd1ca174a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:23.733+0000 7fd1ca174a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:23.747 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:23.747 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:23.747 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:23.747 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:23.747 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:23.748 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:23.752 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:23.753 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:23.753 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:53:23.753 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:23.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:23.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:23.754 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:23.754 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:23.755 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:23.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:23.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:23.756 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:23.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.756 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:23.757 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:23.757 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:23.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:23.757 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:23Z" 2024-05-26T20:53:23.757 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:23.759 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.311444_618.1 2024-05-26T20:53:28.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:28.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:28.760 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:28.815 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:28.815 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:28.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:28.848+0000 7ffa4c1cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:28.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:28.849+0000 7ffa4c1cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:28.863 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.460984_633.1", 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.460984Z", 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:28Z" 2024-05-26T20:53:28.870 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:28.871 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.460984_633.1 2024-05-26T20:53:33.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:33.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:33.872 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:33.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:33.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:33.967+0000 7ff5c2801a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:33.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:33.968+0000 7ff5c2801a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:33.982 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:33.982 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:33.983 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:33.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:33.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:33.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:33Z" 2024-05-26T20:53:33.987 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:33.988 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.370262_622.1 2024-05-26T20:53:38.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:38.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:38.989 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:39.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:39.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:39.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:39.187+0000 7fa61c835a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:39.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:39.191+0000 7fa61c835a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:39.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:39.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:39.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:39.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:39.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:39Z" 2024-05-26T20:53:39.215 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:39.216 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.367915_621.1 2024-05-26T20:53:44.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:44.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:44.218 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:44.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:44.302+0000 7ff29f8c1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:44.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:44.303+0000 7ff29f8c1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:53:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:44Z" 2024-05-26T20:53:44.322 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:44.323 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:53:49.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:49.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:49.325 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:49.377 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:49.377 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:49.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:49.411+0000 7fdc38cd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:49.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:49.412+0000 7fdc38cd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:49.425 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:49.425 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:49.425 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:49.425 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:49.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:49.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:49.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:49Z" 2024-05-26T20:53:49.430 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:49.431 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:53:54.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:54.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:54.433 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:54.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:54.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:54.527+0000 7f7e4e00ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:54.528+0000 7f7e4e00ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.547 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:54.548 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:54.549 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:54.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:54.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:54.552 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:54Z" 2024-05-26T20:53:54.552 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:54.552 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.299030_616.1 2024-05-26T20:53:59.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:59.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:53:59.553 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:53:59.606 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:53:59.606 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:53:59.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:59.638+0000 7f49e7bada80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:59.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:53:59.639+0000 7f49e7bada80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:53:59.652 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:53:59.652 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:53:59.653 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:59.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:59.655 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:53:59.656 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:53:59Z" 2024-05-26T20:53:59.657 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:53:59.658 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:54:04.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:04.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:04.660 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:04.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:04.744+0000 7f492d2c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:04.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:04.745+0000 7f492d2c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:04.758 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:04.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:04.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:04.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:04Z" 2024-05-26T20:54:04.764 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:04.765 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.299030_616.1 2024-05-26T20:54:09.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:09.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:09.767 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:09.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:09.860+0000 7f7055566a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:09.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:09.861+0000 7f7055566a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:09.875 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:09.875 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:09.876 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:09.877 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:09.879 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:09Z" 2024-05-26T20:54:09.880 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:09.881 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:54:14.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:14.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:14.883 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:14.941 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:14.941 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:14.978+0000 7f583a4e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:14.979+0000 7f583a4e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:14.993 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:14.994 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:14.995 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:14.996 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:14.997 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:14.998 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:14.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:14.999 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:14Z" 2024-05-26T20:54:14.999 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:15.000 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.370262_622.1 2024-05-26T20:54:20.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:20.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:20.001 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:20.101 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:20.102 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:20.182+0000 7fb745e9ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:20.185+0000 7fb745e9ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:20.202 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:20.202 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:20.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:20.204 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:20.205 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:20.206 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:20.207 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:20.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:20.209 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:20Z" 2024-05-26T20:54:20.209 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:20.210 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:54:25.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:25.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:25.212 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:25.263 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:25.264 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:25.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:25.294+0000 7f86d2a9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:25.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:25.295+0000 7f86d2a9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:25.308 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:25.309 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:25.310 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:25.311 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:25.312 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:25.313 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:25Z" 2024-05-26T20:54:25.314 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:25.314 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.299030_616.1 2024-05-26T20:54:30.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:30.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:30.316 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:30.375 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:30.376 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:30.410+0000 7fd61e08ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:30.411+0000 7fd61e08ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:30.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:30.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:30.429 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.430 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:30.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:30.431 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:30.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:30.431 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:30Z" 2024-05-26T20:54:30.433 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:30.435 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:54:35.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:35.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:35.436 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:35.495 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:35.495 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:35.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:35.531+0000 7f8ac11eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:35.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:35.532+0000 7f8ac11eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:35.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:35.548 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:35.549 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:35.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:35.551 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:35Z" 2024-05-26T20:54:35.552 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:35.552 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:54:40.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:40.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:40.554 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:40.733+0000 7f9198d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:40.734+0000 7f9198d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:40.758 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:40.758 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:40.758 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:40.758 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:40.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:40.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:54:40.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:40.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:40Z" 2024-05-26T20:54:40.763 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:40.764 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.360000_619.1 2024-05-26T20:54:45.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:45.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:45.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:45.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:45.850+0000 7fe54ac98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:45.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:45.850+0000 7fe54ac98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:45.864 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:45.865 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:45.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:54:45.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:45.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:45Z" 2024-05-26T20:54:45.869 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:45.870 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.311444_618.1 2024-05-26T20:54:50.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:50.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:50.872 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:50.967+0000 7f89516b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:50.968+0000 7f89516b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:50.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:50.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:50.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:50.986 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:50Z" 2024-05-26T20:54:50.987 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:50.988 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.299030_616.1 2024-05-26T20:54:55.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:55.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:54:55.990 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:54:56.043 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:54:56.043 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:54:56.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:56.077+0000 7efe201c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:56.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:54:56.078+0000 7efe201c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:54:56.091 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:56.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:56.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:54:56.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:54:56Z" 2024-05-26T20:54:56.096 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:54:56.097 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.379895_623.1 2024-05-26T20:55:01.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:01.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:01.099 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:01.193+0000 7f7c3ce12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:01.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:01.194+0000 7f7c3ce12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.209 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:01.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:01.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.276118_612.1", 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.276118Z", 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:01.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:01Z" 2024-05-26T20:55:01.214 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:01.215 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.276118_612.1 2024-05-26T20:55:06.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:06.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:06.216 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:06.275 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:06.275 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:06.312+0000 7f8b6d061a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:06.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:06.313+0000 7f8b6d061a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:06.327 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:06.327 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:06.327 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:06.327 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:06.328 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:06.329 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:06.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:06.332 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:06Z" 2024-05-26T20:55:06.333 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:06.335 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.370262_622.1 2024-05-26T20:55:11.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:11.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:11.336 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:11.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:11.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:11.480+0000 7f0fc9e3aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:11.481+0000 7f0fc9e3aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:11.495 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:11.495 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:11.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:11.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:11.498 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:11.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:11Z" 2024-05-26T20:55:11.500 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:11.501 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.367915_621.1 2024-05-26T20:55:16.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:16.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:16.502 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:16.555 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:16.587+0000 7f1d4b7b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:16.587+0000 7f1d4b7b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:16.601 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:16.601 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:16.601 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:16.601 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:16.601 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:16.602 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:16.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:16.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:16.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:16Z" 2024-05-26T20:55:16.606 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:16.607 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:55:21.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:21.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:21.608 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:21.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:21.696+0000 7f6c24161a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:21.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:21.697+0000 7f6c24161a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:21.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:21.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.715 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:21.716 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:21.716 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:21.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:21.716 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:21Z" 2024-05-26T20:55:21.716 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:21.717 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.296023_614.1 2024-05-26T20:55:26.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:26.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:26.718 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:26.772 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:26.772 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:26.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:26.804+0000 7f9edb67aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:26.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:26.806+0000 7f9edb67aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:26.819 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:26.819 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:26.819 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:26.820 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:26.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.539696_639.1", 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:26.822 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.539696Z", 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:26.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:26Z" 2024-05-26T20:55:26.824 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:26.825 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.539696_639.1 2024-05-26T20:55:31.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:31.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:31.826 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:31.922+0000 7f7137c45a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:31.923+0000 7f7137c45a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:31.938 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:31Z" 2024-05-26T20:55:31.944 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:31.945 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:55:36.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:36.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:36.946 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:37.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:37.043+0000 7fd94b9eba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:37.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:37.044+0000 7fd94b9eba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:37.058 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:37.058 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:37.058 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:37.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:37.060 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:37.061 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:37.062 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:37Z" 2024-05-26T20:55:37.063 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:37.064 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.360000_619.1 2024-05-26T20:55:42.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:42.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:42.066 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:42.119 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:42.119 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:42.153+0000 7f12f03fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:42.154+0000 7f12f03fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:42Z" 2024-05-26T20:55:42.173 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:42.174 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.367915_621.1 2024-05-26T20:55:47.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:47.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:47.175 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:47.265+0000 7f2c9c24ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:47.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:47.266+0000 7f2c9c24ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:47.281 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:47.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:55:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:47.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:47Z" 2024-05-26T20:55:47.286 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:47.287 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:55:52.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:52.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:52.289 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:52.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:52.347 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:52.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:52.382+0000 7fae91207a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:52.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:52.383+0000 7fae91207a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:52Z" 2024-05-26T20:55:52.403 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:52.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.645969_653.1 2024-05-26T20:55:57.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:57.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:55:57.405 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:55:57.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:55:57.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:55:57.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:57.593+0000 7fa94c816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:55:57.595+0000 7fa94c816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:55:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:57.627 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:55:57.629 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:55:57.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:55:57.631 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:55:57Z" 2024-05-26T20:55:57.631 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:55:57.632 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:56:02.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:02.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:02.633 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:02.684 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:02.684 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:02.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:02.715+0000 7f5179c5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:02.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:02.716+0000 7f5179c5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:02.730 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T20:56:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:02Z" 2024-05-26T20:56:02.735 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:02.736 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.395458_627.1 2024-05-26T20:56:07.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:07.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:07.738 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:07.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:07.833+0000 7f9b4411ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:07.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:07.834+0000 7f9b4411ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:07.848 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:56:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:07Z" 2024-05-26T20:56:07.854 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:07.854 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:56:12.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:12.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:12.856 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:12.915 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:12.915 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:12.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:12.949+0000 7f7517421a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:12.950+0000 7f7517421a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:56:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:12.970 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:12.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:12.971 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:12Z" 2024-05-26T20:56:12.971 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:12.972 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.360000_619.1 2024-05-26T20:56:17.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:17.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:17.973 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:18.032 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:18.032 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:18.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:18.067+0000 7f4a81211a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:18.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:18.068+0000 7f4a81211a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:18.083 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:18.083 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:18.083 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:18.083 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:18.083 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:18.084 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:18.085 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:18.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:18.087 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:18Z" 2024-05-26T20:56:18.088 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:18.089 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.360000_619.1 2024-05-26T20:56:23.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:23.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:23.091 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:23.150 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:23.150 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:23.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:23.187+0000 7f6dc0d9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:23.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:23.188+0000 7f6dc0d9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:23.202 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:23.203 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:23.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:23.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:23.206 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:23Z" 2024-05-26T20:56:23.207 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:23.208 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.296898_615.1 2024-05-26T20:56:28.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:28.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:28.210 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:28.304 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:28.304 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:28.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:28.343+0000 7f327987ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:28.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:28.344+0000 7f327987ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:28.359 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:28.359 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:28.359 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:28.359 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:28.361 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:28.362 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:28.363 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:28Z" 2024-05-26T20:56:28.364 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:28.365 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.309042_617.1 2024-05-26T20:56:33.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:33.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:33.367 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:33.418 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:33.418 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:33.449+0000 7faea6e6aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:33.450+0000 7faea6e6aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:33.464 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:33.464 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:33.464 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:33.464 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:33.465 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:33.466 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.400971_630.1", 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:33.467 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.400971Z", 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:33.468 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:33Z" 2024-05-26T20:56:33.469 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:33.470 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.400971_630.1 2024-05-26T20:56:38.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:38.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:38.471 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:38.525 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:38.525 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:38.557+0000 7f74bc2cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:38.558+0000 7f74bc2cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:38Z" 2024-05-26T20:56:38.577 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:38.578 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.296023_614.1 2024-05-26T20:56:43.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:43.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:43.579 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:43.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:43.671+0000 7fc2c5d70a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:43.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:43.672+0000 7fc2c5d70a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:43Z" 2024-05-26T20:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:43.697 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:56:48.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:48.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:48.699 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:48.794+0000 7f4db734ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:48.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:48.795+0000 7f4db734ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:48.809 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:48.810 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:48.811 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:48.812 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:48.813 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:48.814 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:48.815 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:48.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:48.815 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:48Z" 2024-05-26T20:56:48.815 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:48.816 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.370262_622.1 2024-05-26T20:56:53.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:53.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:56:53.817 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:56:53.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:53.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:53.910+0000 7fbd9a3efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:56:53.911+0000 7fbd9a3efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:53.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:56:53.928 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.293638_613.1", 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.293638Z", 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:56:53.929 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:56:53.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:56:53.931 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:56:53Z" 2024-05-26T20:56:53.931 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:56:53.932 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716756705.971680_773.1 target=1_1716756067.293638_613.1 2024-05-26T20:56:58.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-05-26T20:56:58.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:56:58.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:56:58.935 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:56:58.991 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:58.991 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:59.013 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone2", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "test-zone2.rgw.control", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "test-zone2.rgw.log", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-05-26T20:56:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "WB0LE5I0BAFHL536", 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "TUtFQih6d8vJyiEWzurVtHMAn8IwKknK" 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-05-26T20:56:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-05-26T20:56:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-05-26T20:56:59.017 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:56:59.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:56:59.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T20:56:59.018 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T20:56:59.068 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:56:59.068 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:56:59.108 INFO:teuthology.orchestra.run.smithi137.stderr:Sending period to new master zone 7e72b8a9-5d69-4e33-98d3-490830e32745 2024-05-26T20:56:59.130 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:56:59.130 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:59.130 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 4, 2024-05-26T20:56:59.130 INFO:teuthology.orchestra.run.smithi137.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": [], 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "period_map": { 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "zonegroups": [ 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "test-zonegroup", 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zonegroup", 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": true, 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.131 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone3", 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:56:59.132 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:56:59.133 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone2", 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": false, 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-05-26T20:56:59.134 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-zone1", 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-05-26T20:56:59.135 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-05-26T20:56:59.136 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-05-26T20:56:59.137 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: "short_zone_ids": [ 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3864594665 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T20:56:59.138 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3171627794 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3772860827 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T20:56:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "period_config": { 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_quota": { 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "user_quota": { 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-05-26T20:56:59.143 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "user_ratelimit": { 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_ratelimit": { 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:56:59.144 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "anonymous_ratelimit": { 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T20:56:59.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_name": "test-realm", 2024-05-26T20:56:59.146 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:56:59.146 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:56:59.147 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-05-26T20:57:59.148 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-48 2024-05-26T20:57:59.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:57:59.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:57:59.195 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:57:59.319 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:57:59.319 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:57:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:57:59.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:57:59.172867Z" 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:57:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T20:57:59.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T20:57:59.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:57:59.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:57:59.397 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:57:59.454 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:57:59.454 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:57:59.485 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:57:59.486+0000 7ff7657c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:57:59.486 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:57:59.487+0000 7ff7657c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:57:59.498 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:57:59.499 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:57:59.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:57:59.501 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:57:59.502 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:57:59.503 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:57:59Z" 2024-05-26T20:57:59.504 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:57:59.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:04.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:04.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:04.506 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:04.563 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:04.564 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:04.596 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:04.597+0000 7ff290ae7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:04.597 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:04.598+0000 7ff290ae7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:04.609 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:04.610 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:04.611 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:04.612 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:04.613 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:04Z" 2024-05-26T20:58:04.614 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:04.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:09.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:09.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:09.617 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:09.673 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:09.674 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:09.705 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:09.706+0000 7f88c1c57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:09.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:09.707+0000 7f88c1c57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:09.718 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:09.718 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:09.721 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:09.722 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:09Z" 2024-05-26T20:58:09.723 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:09.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:14.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:14.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:14.726 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:14.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:14.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:14.815 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:14.817+0000 7f9dc7a53a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:14.816 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:14.818+0000 7f9dc7a53a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:14.829 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:14.829 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:14.829 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:14.830 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:14Z" 2024-05-26T20:58:14.835 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:14.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:19.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:19.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:19.837 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:19.896 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:19.896 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:19.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:19.927+0000 7eff18af7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:19.928 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:19.928+0000 7eff18af7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:19.940 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:19Z" 2024-05-26T20:58:19.945 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:19.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:24.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:24.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:24.948 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:25.006 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:25.007 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:25.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:25.038+0000 7f5a77bb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:25.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:25.039+0000 7f5a77bb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:25.051 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:25.053 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:25.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:25.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:25Z" 2024-05-26T20:58:25.056 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:25.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:30.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:30.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:30.059 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:30.116 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:30.116 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:30.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:30.148+0000 7fd32a8aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:30.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:30.148+0000 7fd32a8aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:30.161 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:30.161 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:30.161 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:30.161 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:30.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:30.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:30.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:30.165 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:30.166 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:30Z" 2024-05-26T20:58:30.167 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:30.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:35.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:35.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:35.170 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:35.227 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:35.227 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:35.258 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:35.258+0000 7f39767efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:35.259 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:35.259+0000 7f39767efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:35.272 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:35.272 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:35.272 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:35.273 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:35.275 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:35.276 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:35Z" 2024-05-26T20:58:35.277 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:35.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:40.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:40.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:40.279 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:40.337 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:40.337 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:40.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:40.368+0000 7f4886258a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:40.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:40.369+0000 7f4886258a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:40.381 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:40Z" 2024-05-26T20:58:40.386 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:40.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:45.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:45.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:45.389 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:45.446 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:45.447 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:45.478 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:45.478+0000 7f5f57ae8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:45.479 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:45.479+0000 7f5f57ae8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:45.495 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.496 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:45.497 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:45.497 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:45.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:45.497 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:45Z" 2024-05-26T20:58:45.497 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:45.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:50.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:50.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:50.499 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:50.557 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:50.557 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:50.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:50.588+0000 7f5cdeca3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:50.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:50.589+0000 7f5cdeca3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:50.601 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:50Z" 2024-05-26T20:58:50.606 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:50.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:58:55.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:55.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:58:55.609 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:58:55.666 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:58:55.666 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:58:55.697 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:55.698+0000 7f5e3239ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:55.698 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:58:55.699+0000 7f5e3239ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:55.711 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:58:55.712 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:58:55.713 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:58:55.714 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:58:55.715 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:58:55.716 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:58:55.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:58:55.716 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:58:55Z" 2024-05-26T20:58:55.716 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:58:55.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756705.971680_773.1 2024-05-26T20:59:00.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:00.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:00.718 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:00.774 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:00.774 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:00.805 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:59:00.806+0000 7f31fc059a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:00.805 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T20:59:00.807+0000 7f31fc059a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:00.818 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T20:59:00.818 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T20:59:00.818 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T20:59:00.818 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T20:59:00.818 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:00.819 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:59:00.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T20:59:00.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T20:59:00Z" 2024-05-26T20:59:00.823 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T20:59:00.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T20:59:00.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T20:59:00.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:00.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:00.825 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:00.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:00.930+0000 7fbd06e81a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:00.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:00.931+0000 7fbd06e81a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:00.944 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:00.944 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:00.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:00.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.477787_636.1", 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:00.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.477787Z", 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:00.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:00Z" 2024-05-26T20:59:00.949 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:00.950 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.477787_636.1 2024-05-26T20:59:05.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:05.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:05.951 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:06.038+0000 7fddff6c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:06.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:06.039+0000 7fddff6c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:06.052 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:06.052 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:06.053 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:06.054 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:06.055 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:06.056 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.058 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:06.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:06.059 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:06.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:06.059 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:06Z" 2024-05-26T20:59:06.059 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:06.060 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T20:59:11.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:11.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:11.062 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:11.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:11.150+0000 7fa934fbaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:11.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:11.151+0000 7fa934fbaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:11.165 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:11Z" 2024-05-26T20:59:11.171 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:11.172 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T20:59:16.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:16.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:16.174 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:16.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:16.265+0000 7fa285307a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:16.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:16.266+0000 7fa285307a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:16.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:16.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:16.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:16.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:16Z" 2024-05-26T20:59:16.285 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:16.286 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.379895_623.1 2024-05-26T20:59:21.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:21.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:21.288 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:21.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:21.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:21.382+0000 7f301b36ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:21.383+0000 7f301b36ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:21.398 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:21.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:21.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:21.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:21Z" 2024-05-26T20:59:21.402 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:21.403 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T20:59:26.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:26.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:26.405 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:26.463 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:26.463 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:26.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:26.498+0000 7efc7c8eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:26.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:26.499+0000 7efc7c8eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:26.514 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:26.515 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:26.516 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:26.517 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:26.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:26.520 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:26Z" 2024-05-26T20:59:26.520 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:26.521 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T20:59:31.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:31.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:31.522 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:31.616 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:31.617 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:31.704+0000 7f3924d64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:31.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:31.710+0000 7f3924d64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:31Z" 2024-05-26T20:59:31.742 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:31.743 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T20:59:36.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:36.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:36.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:36.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:36.832+0000 7fcbf4dd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:36.833+0000 7fcbf4dd1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:36.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:36.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:36.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:36.851 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:36Z" 2024-05-26T20:59:36.852 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:36.853 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.382628_624.1 2024-05-26T20:59:41.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:41.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:41.854 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:41.938+0000 7fa82f2aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:41.939+0000 7fa82f2aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.953 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:41.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:41.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T20:59:41.956 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:41.957 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:41Z" 2024-05-26T20:59:41.958 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:41.959 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T20:59:46.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:46.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:46.961 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:47.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:47.049+0000 7f79a3486a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:47.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:47.050+0000 7f79a3486a80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:47Z" 2024-05-26T20:59:47.070 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:47.071 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T20:59:52.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:52.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:52.072 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:52.132 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:52.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:52.167+0000 7faad14fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:52.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:52.168+0000 7faad14fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.183 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:52.184 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:52.185 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:52.186 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:52.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:52.188 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:52Z" 2024-05-26T20:59:52.188 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:52.189 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T20:59:57.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:57.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T20:59:57.190 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T20:59:57.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T20:59:57.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T20:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:57.285+0000 7f7cc012ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T20:59:57.286+0000 7f7cc012ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T20:59:57.301 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T20:59:57.301 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T20:59:57.301 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T20:59:57.301 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T20:59:57.301 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:57.302 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T20:59:57.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.460984_633.1", 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.460984Z", 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:57.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:57.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T20:59:57Z" 2024-05-26T20:59:57.306 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T20:59:57.307 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.460984_633.1 2024-05-26T21:00:02.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:02.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:02.308 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:02.366 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:02.366 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:02.402+0000 7fc406910a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:02.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:02.404+0000 7fc406910a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:02.420 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:02.421 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:02Z" 2024-05-26T21:00:02.424 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:02.425 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:00:07.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:07.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:07.427 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:07.486 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:07.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:07.549+0000 7eff0ada9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:07.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:07.553+0000 7eff0ada9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:07.626 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:07.626 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:07.628 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:07.629 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:07.630 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:07.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:07.632 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:07Z" 2024-05-26T21:00:07.632 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:07.633 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.382628_624.1 2024-05-26T21:00:12.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:12.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:12.634 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:12.686 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:12.686 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:12.718+0000 7f2c974cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:12.719+0000 7f2c974cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:12.732 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:12.732 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:12.732 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:12.732 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:12.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:12.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.385899_626.1", 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.385899Z", 2024-05-26T21:00:12.735 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:12.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:12Z" 2024-05-26T21:00:12.737 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:12.738 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.385899_626.1 2024-05-26T21:00:17.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:17.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:17.742 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:17.831+0000 7f8910acea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:17.832+0000 7f8910acea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:17.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:17.850 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:17Z" 2024-05-26T21:00:17.850 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:17.851 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:00:22.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:22.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:22.852 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:22.909 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:22.909 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:22.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:22.943+0000 7f3b50ebca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:22.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:22.944+0000 7f3b50ebca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:22.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:22.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.963 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:22.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:22.964 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:22.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:22.964 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:22Z" 2024-05-26T21:00:22.964 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:22.965 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:00:27.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:27.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:27.966 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:28.025 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:28.025 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:28.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:28.062+0000 7f0c4cc18a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:28.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:28.063+0000 7f0c4cc18a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:28.078 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:28.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:28.082 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:28Z" 2024-05-26T21:00:28.083 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:28.084 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:00:33.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:33.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:33.086 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:33.144 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:33.144 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:33.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:33.179+0000 7f2d01f65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:33.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:33.180+0000 7f2d01f65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:33.194 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:00:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:33Z" 2024-05-26T21:00:33.199 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:33.200 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:00:38.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:38.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:38.202 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:38.314 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:38.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:38.375+0000 7f9215052a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:38.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:38.378+0000 7f9215052a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:38.392 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:38.392 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:38.392 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:38.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:38.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:38.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:38Z" 2024-05-26T21:00:38.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:38.398 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:00:43.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:43.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:43.400 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:43.452 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:43.452 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:43.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:43.484+0000 7f0d187ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:43.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:43.485+0000 7f0d187ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:43Z" 2024-05-26T21:00:43.503 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:43.504 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:00:48.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:48.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:48.506 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:48.590+0000 7f683fc68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:48.590+0000 7f683fc68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.604 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:48.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:48.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:48.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:48Z" 2024-05-26T21:00:48.609 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:48.610 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.457698_631.1 2024-05-26T21:00:53.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:53.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:53.611 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:53.662 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:53.662 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:53.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:53.694+0000 7f7088ad7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:53.695+0000 7f7088ad7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:53.709 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:53.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:53.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:53.712 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:53.713 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:53.714 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:53.715 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:53.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:53.715 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:53Z" 2024-05-26T21:00:53.715 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:53.716 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.645969_653.1 2024-05-26T21:00:58.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:58.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:00:58.718 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:00:58.771 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:00:58.771 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:00:58.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:58.803+0000 7feab9220a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:58.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:00:58.804+0000 7feab9220a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:58.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:00:58.820 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.474724_635.1", 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:58.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.474724Z", 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:00:58.822 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:00:58.823 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:00:58.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:00:58.824 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:00:58Z" 2024-05-26T21:00:58.824 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:00:58.825 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.474724_635.1 2024-05-26T21:01:03.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:03.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:03.826 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:03.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:03.922+0000 7fc00dcb0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:03.923+0000 7fc00dcb0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:03.938 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:03.939 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:03.940 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:03.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:03.943 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:03Z" 2024-05-26T21:01:03.944 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:03.945 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:01:08.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:08.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:08.947 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:09.006 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:09.006 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:09.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:09.042+0000 7fbce96b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:09.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:09.043+0000 7fbce96b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:09.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:09.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:09.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:09Z" 2024-05-26T21:01:09.063 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:09.064 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:01:14.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:14.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:14.065 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:14.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:14.161+0000 7f008b025a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:14.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:14.162+0000 7f008b025a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:14.178 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:14.179 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:14.180 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:01:14.181 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:14.182 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:14Z" 2024-05-26T21:01:14.183 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:14.184 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:01:19.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:19.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:19.185 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:19.242 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:19.242 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:19.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:19.286+0000 7fbba1993a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:19.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:19.311+0000 7fbba1993a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:19.383 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:19.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:19.385 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:19.386 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:19.387 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:19Z" 2024-05-26T21:01:19.388 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:19.389 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.395458_627.1 2024-05-26T21:01:24.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:24.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:24.391 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:24.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:24.477+0000 7f482a4b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:24.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:24.478+0000 7f482a4b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:24.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:24.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:24.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.397876_628.1", 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.397876Z", 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:24.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:24.495 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:24.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:24.496 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:24Z" 2024-05-26T21:01:24.496 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:24.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.397876_628.1 2024-05-26T21:01:29.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:29.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:29.498 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:29.583+0000 7fb6ebd09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:29.584+0000 7fb6ebd09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.598 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:29.600 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:29.601 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:29.602 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:29Z" 2024-05-26T21:01:29.603 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:29.605 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:01:34.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:34.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:34.606 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:34.694+0000 7f43f0cb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:34.695+0000 7f43f0cb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:34.709 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:34.709 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:34.709 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:34.709 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:34.709 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:34.710 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:34.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:34.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:34.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:34Z" 2024-05-26T21:01:34.714 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:34.718 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.395458_627.1 2024-05-26T21:01:39.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:39.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:39.719 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:39.774 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:39.775 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:39.808+0000 7f0b6f219a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:39.809+0000 7f0b6f219a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:39.823 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:39.824 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:39.825 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:39.826 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.460055_632.1", 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.460055Z", 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:39.827 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:39.828 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:39Z" 2024-05-26T21:01:39.829 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:39.829 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.460055_632.1 2024-05-26T21:01:44.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:44.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:44.831 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:44.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:44.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:44.926+0000 7ff99c722a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:44.927+0000 7ff99c722a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:01:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:44Z" 2024-05-26T21:01:44.952 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:44.953 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:01:49.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:49.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:49.955 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:50.113+0000 7f248dcdaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:50.119+0000 7f248dcdaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:50.173 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:50.178 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:50.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:50.180 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.181 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:50.182 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:50.182 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:50.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:50.182 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:50Z" 2024-05-26T21:01:50.182 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:50.183 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:01:55.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:55.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:01:55.184 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:01:55.236 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:01:55.236 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:01:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:55.266+0000 7fcba0280a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:01:55.267+0000 7fcba0280a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:01:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:01:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:01:55.287 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:01:55.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:01:55.287 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:01:55Z" 2024-05-26T21:01:55.287 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:01:55.288 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:02:00.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:00.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:00.289 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:00.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:00.383+0000 7f43bb73fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:00.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:00.384+0000 7f43bb73fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:00.398 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:00.398 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:00.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:00.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.526780_637.1", 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.526780Z", 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:00Z" 2024-05-26T21:02:00.403 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:00.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.526780_637.1 2024-05-26T21:02:05.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:05.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:05.406 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:05.465 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:05.466 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:05.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:05.502+0000 7f5a41a35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:05.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:05.503+0000 7f5a41a35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:05.519 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:05.521 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:05.522 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:05.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:05.523 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:05Z" 2024-05-26T21:02:05.523 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:05.524 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:02:10.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:10.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:10.525 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:10.585 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:10.585 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:10.620+0000 7efcf8930a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:10.621+0000 7efcf8930a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:10.637 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:10.637 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:10.637 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:10.637 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:10.638 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:10.639 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.457698_631.1", 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.457698Z", 2024-05-26T21:02:10.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:10.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:10Z" 2024-05-26T21:02:10.642 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:10.643 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.457698_631.1 2024-05-26T21:02:15.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:15.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:15.645 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:15.703 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:15.704 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:15.739+0000 7fb916b76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:15.740+0000 7fb916b76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:15.755 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:15.755 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:15.756 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:15.757 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:15.758 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.760 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:15.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:15.761 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:15.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:15.761 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:15Z" 2024-05-26T21:02:15.761 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:15.762 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:02:20.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:20.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:20.763 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:20.864 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:20.864 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:20.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:20.915+0000 7efcf5864a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:20.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:20.916+0000 7efcf5864a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:20.932 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:20.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:20.933 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:20.933 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:20.933 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:20.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:20.934 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:20.934 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:20.934 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:20.934 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:20.935 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:02:20.936 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:20.937 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:20.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:20.939 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:20Z" 2024-05-26T21:02:20.939 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:20.940 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:02:25.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:25.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:25.941 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:25.995 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:25.995 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:26.026+0000 7ff741f46a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:26.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:26.026+0000 7ff741f46a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:26.042 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:26.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:26.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:26.045 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:26Z" 2024-05-26T21:02:26.045 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:26.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.395458_627.1 2024-05-26T21:02:31.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:31.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:31.047 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:31.099 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:31.099 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:31.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:31.132+0000 7f502b949a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:31.133+0000 7f502b949a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:31.146 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:31.146 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:31.146 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:31.146 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:31.146 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:02:31.149 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:31.150 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:31Z" 2024-05-26T21:02:31.151 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:31.152 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:02:36.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:36.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:36.154 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:36.247+0000 7f22d1911a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:36.248+0000 7f22d1911a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:36.262 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:36.262 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:36.262 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:36.262 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:36.263 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:36.264 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:36.265 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:02:36.266 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:36.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:36Z" 2024-05-26T21:02:36.268 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:36.270 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:02:41.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:41.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:41.271 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:41.360+0000 7fe244fb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:41.361+0000 7fe244fb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:41.375 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:41.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:41.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:41.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.551890_641.1", 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.551890Z", 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:41Z" 2024-05-26T21:02:41.381 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:41.382 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.551890_641.1 2024-05-26T21:02:46.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:46.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:46.384 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:46.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:46.479+0000 7f9c80fb1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:46.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:46.480+0000 7f9c80fb1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:46.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:46.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:46.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:46.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:46.499 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:46Z" 2024-05-26T21:02:46.500 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:46.501 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.382628_624.1 2024-05-26T21:02:51.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:51.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:51.503 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:51.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:51.598+0000 7f178dd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:51.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:51.599+0000 7f178dd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:51.646 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:51.646 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:51.651 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:51.651 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:51.651 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:51.651 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:51.652 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:51.652 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:51.652 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:51.652 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:51.652 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:51.654 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:51.654 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:02:51.654 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:51.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:51.654 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:51.655 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:02:51.655 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:51.655 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:51.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:51.656 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:51.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:51.657 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:51.657 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:51.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:51.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:51.658 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:51.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.658 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:51.658 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:51.658 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:51.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:51.659 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:51Z" 2024-05-26T21:02:51.659 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:51.660 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:02:56.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:56.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:02:56.662 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:02:56.711 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:02:56.712 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:02:56.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:56.743+0000 7f9a5b098a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:56.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:02:56.744+0000 7f9a5b098a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:02:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:02:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:02:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:02:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:02:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:02:56Z" 2024-05-26T21:02:56.763 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:02:56.763 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:03:01.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:01.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:01.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:01.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:01.855+0000 7fc43e333a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:01.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:01.856+0000 7fc43e333a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:01.870 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:01.870 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:01.870 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:01.870 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:01.870 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:01.871 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:01.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.383646_625.1", 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.383646Z", 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:01.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:01.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:01Z" 2024-05-26T21:03:01.875 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:01.877 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.383646_625.1 2024-05-26T21:03:06.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:06.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:06.878 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:06.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:06.969+0000 7f08724f1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:06.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:06.970+0000 7f08724f1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:06.985 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:06.985 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:06.985 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:06.985 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:06.985 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:06.986 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:06.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:06.988 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:06.989 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:06.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:06.991 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.992 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:06.993 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:06.993 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:06.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:06.993 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:06Z" 2024-05-26T21:03:06.993 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:06.994 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:03:11.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:11.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:11.996 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:12.054 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:12.055 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:12.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:12.090+0000 7f04cc7c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:12.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:12.091+0000 7f04cc7c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:12.106 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:12.106 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:12.106 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:12.106 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:12.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:12.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:12.109 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:12.110 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:12.111 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:12.114 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:12.114 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:12.114 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:12.114 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:12.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:12.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:12Z" 2024-05-26T21:03:12.116 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:12.118 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:03:17.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:17.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:17.119 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:17.178 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:17.179 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:17.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:17.214+0000 7f0a63d07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:17.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:17.215+0000 7f0a63d07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:17.230 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:17.230 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:17.230 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:17.231 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:17.232 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:17.234 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:17Z" 2024-05-26T21:03:17.235 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:17.236 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:03:22.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:22.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:22.238 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:22.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:22.443+0000 7f5e823c1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:22.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:22.447+0000 7f5e823c1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:22Z" 2024-05-26T21:03:22.486 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:22.487 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:03:27.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:27.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:27.488 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:27.540 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:27.540 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:27.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:27.571+0000 7f0aac6aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:27.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:27.572+0000 7f0aac6aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:27.586 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:27.586 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:27.586 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:27.587 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:27.588 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:27.589 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:27.590 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:27Z" 2024-05-26T21:03:27.591 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:27.592 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:03:32.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:32.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:32.594 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:32.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:32.683+0000 7f5409515a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:32.684+0000 7f5409515a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:32.697 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:32.698 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:32.699 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:32.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:32.701 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:32Z" 2024-05-26T21:03:32.702 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:32.703 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:03:37.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:37.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:37.704 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:37.756 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:37.756 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:37.788+0000 7fc5829dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:37.789+0000 7fc5829dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:37.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:37.804 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:03:37.805 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:37.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:37Z" 2024-05-26T21:03:37.807 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:37.808 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:03:42.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:42.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:42.810 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:42.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:42.893+0000 7f513fb40a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:42.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:42.894+0000 7f513fb40a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:42.908 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:03:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:42Z" 2024-05-26T21:03:43.165 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:43.166 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:03:48.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:48.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:48.167 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:48.264+0000 7fc528aefa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:48.265+0000 7fc528aefa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:48.279 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:48.279 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:48.280 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:48.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:48.282 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:48.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:48Z" 2024-05-26T21:03:48.284 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:48.285 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:03:53.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:53.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:53.287 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:53.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:53.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:53.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:53.382+0000 7f71cad12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:53.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:53.383+0000 7f71cad12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:53.397 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:53.398 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:53.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:53.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.474724_635.1", 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.474724Z", 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:53.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:53.402 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:53Z" 2024-05-26T21:03:53.403 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:53.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.474724_635.1 2024-05-26T21:03:58.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:58.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:03:58.405 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:03:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:03:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:03:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:58.613+0000 7f054bda2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:03:58.614+0000 7f054bda2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:03:58.629 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:03:58.629 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:03:58.629 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:03:58.629 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:58.630 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:58.631 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:03:58.632 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:03:58.633 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:03:58Z" 2024-05-26T21:03:58.634 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:03:58.635 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:04:03.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:03.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:03.637 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:03.688 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:03.688 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:03.720+0000 7fe19a28ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:03.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:03.722+0000 7fe19a28ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:03.735 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:03Z" 2024-05-26T21:04:03.740 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:03.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:04:08.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-05-26T21:04:08.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:04:08.838 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:04:08.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:57:59.172867Z" 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:08.839 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:04:08.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:08.840 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:04:08.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:04:08.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.841 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:08.899 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:08.899 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:08.930 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:04:08.930+0000 7fa7d0262a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:08.931 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:04:08.932+0000 7fa7d0262a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:04:08.945 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:04:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:04:08.947 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:04:08.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:04:08Z" 2024-05-26T21:04:08.949 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:04:08.950 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:04:08.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:04:08.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:08.950 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:09.001 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:09.001 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:09.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:09.032+0000 7fdac9fbda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:09.033+0000 7fdac9fbda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:09.046 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:09.047 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:09.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.400971_630.1", 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.400971Z", 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:09.049 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.050 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:09.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:09.051 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:09.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:09.051 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:09Z" 2024-05-26T21:04:09.051 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:09.052 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.400971_630.1 2024-05-26T21:04:14.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:14.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:14.053 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:14.112 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:14.112 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:14.148+0000 7f9a3d156a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:14.149+0000 7f9a3d156a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:14.164 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:14.164 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:14.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:14.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:14.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:14.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:14Z" 2024-05-26T21:04:14.169 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:14.170 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.362486_620.1 2024-05-26T21:04:19.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:19.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:19.172 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:19.229 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:19.229 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:19.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:19.354+0000 7f2a3c690a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:19.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:19.358+0000 7f2a3c690a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:19.391 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:19.392 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:19.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:19.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:19Z" 2024-05-26T21:04:19.396 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:19.397 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:04:24.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:24.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:24.399 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:24.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:24.585+0000 7fc18f7d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:24.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:24.586+0000 7fc18f7d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:24.601 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:24.601 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:24.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:24.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:24.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:24.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:24Z" 2024-05-26T21:04:24.606 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:24.607 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:04:29.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:29.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:29.609 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:29.660 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:29.660 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:29.691+0000 7f6c94e25a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:29.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:29.692+0000 7f6c94e25a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:29.705 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:29.705 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:29.705 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:29.706 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:29.707 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:29.709 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.710 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:29.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:29.711 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:29.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:29.711 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:29Z" 2024-05-26T21:04:29.711 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:29.712 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:04:34.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:34.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:34.713 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:34.766 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:34.766 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:34.799+0000 7f88b6960a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:34.800+0000 7f88b6960a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:34.814 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:34.815 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:34Z" 2024-05-26T21:04:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:34.820 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:04:39.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:39.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:39.821 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:39.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:39.875 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:39.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:39.909+0000 7f895904ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:39.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:39.910+0000 7f895904ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.928 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:39.929 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:39.929 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:39.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:39.929 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:39Z" 2024-05-26T21:04:39.929 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:39.930 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:04:44.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:44.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:44.932 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:44.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:44.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:45.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:45.022+0000 7f653e5d9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:45.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:45.023+0000 7f653e5d9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:45.037 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:45Z" 2024-05-26T21:04:45.043 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:45.044 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:04:50.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:50.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:50.046 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:50.140+0000 7fa71978aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:50.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:50.141+0000 7fa71978aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:50.156 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:50.157 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:50.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:50.161 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:50.161 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:50.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:50.161 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:50Z" 2024-05-26T21:04:50.161 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:50.162 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:04:55.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:55.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:04:55.164 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:04:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:04:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:04:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:55.262+0000 7fc71dfc6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:04:55.263+0000 7fc71dfc6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:04:55.278 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:55.279 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:55.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:55.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:04:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:55.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:55.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:04:55.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.285 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:04:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:04:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:04:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:04:55.286 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:04:55Z" 2024-05-26T21:04:55.286 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:04:55.287 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:05:00.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:00.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:00.288 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:00.385+0000 7f8121cd6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:00.386+0000 7f8121cd6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:00.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:00.404 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.405 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:00.406 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:00.406 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:00.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:00.406 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:00Z" 2024-05-26T21:05:00.406 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:00.407 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:05:05.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:05.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:05.409 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:05.607+0000 7f992c871a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:05.610+0000 7f992c871a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:05.636 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:05.637 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:05.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:05Z" 2024-05-26T21:05:05.641 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:05.642 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:05:10.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:10.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:10.644 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:10.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:10.808+0000 7f14df5f2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:10.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:10.809+0000 7f14df5f2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:10.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:10.823 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:10.825 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:10.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:10.827 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:10Z" 2024-05-26T21:05:10.827 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:10.828 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:05:15.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:15.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:15.829 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:15.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:15.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:15.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:15.991+0000 7feb52667a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:15.992+0000 7feb52667a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:16.007 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:16.007 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:16.007 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:16.007 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:16.007 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:16.008 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:16.009 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:16.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:05:16.011 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:16.012 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:16.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:16.014 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:15Z" 2024-05-26T21:05:16.015 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:16.016 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:05:21.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:21.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:21.016 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:21.112 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:21.112 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:21.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:21.149+0000 7f92fc5c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:21.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:21.150+0000 7f92fc5c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:21.164 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:21.164 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:21.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:21.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:21.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.539696_639.1", 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.539696Z", 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:21.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:21Z" 2024-05-26T21:05:21.170 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:21.170 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.539696_639.1 2024-05-26T21:05:26.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:26.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:26.172 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:26.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:26.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:26.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:26.284+0000 7f8102c6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:26.285+0000 7f8102c6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:26.300 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:26.300 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:26.301 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:26.303 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:26.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:26.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:26Z" 2024-05-26T21:05:26.306 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:26.307 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:05:31.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:31.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:31.308 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:31.440 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:31.441 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:31.476+0000 7f57ea4c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:31.477+0000 7f57ea4c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:31.491 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:31.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:31.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:31.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:31.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:31.496 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:31Z" 2024-05-26T21:05:31.496 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:31.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.362486_620.1 2024-05-26T21:05:36.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:36.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:36.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:36.586+0000 7f27de1a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:36.587+0000 7f27de1a6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:36.601 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:36.602 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:36.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:36.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:36.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:36Z" 2024-05-26T21:05:36.606 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:36.607 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:05:41.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:41.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:41.608 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:41.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:41.928+0000 7fd312933a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:41.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:41.929+0000 7fd312933a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:41.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:41.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:41.948 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:41.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:41.950 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:41Z" 2024-05-26T21:05:41.950 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:41.951 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.379895_623.1 2024-05-26T21:05:46.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:46.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:46.952 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:47.048+0000 7f1d6bfcca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:47.049+0000 7f1d6bfcca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:47.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:47.065 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:47.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.382628_624.1", 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.382628Z", 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:47.067 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:47.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:47Z" 2024-05-26T21:05:47.069 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:47.070 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.382628_624.1 2024-05-26T21:05:52.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:52.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:52.072 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:52.204+0000 7fda55e31a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:52.205+0000 7fda55e31a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:52.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:52.225 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:52Z" 2024-05-26T21:05:52.225 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:52.225 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:05:57.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:57.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:05:57.227 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:05:57.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:05:57.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:05:57.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:57.309+0000 7f103329ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:05:57.310+0000 7f103329ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:05:57.324 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:05:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:05:57Z" 2024-05-26T21:05:57.328 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:05:57.329 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.362486_620.1 2024-05-26T21:06:02.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:02.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:02.331 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:02.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:02.412+0000 7fc554e33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:02.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:02.414+0000 7fc554e33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:02.428 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:02.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:02.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:02.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.557590_644.1", 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.557590Z", 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:02Z" 2024-05-26T21:06:02.433 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:02.435 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.557590_644.1 2024-05-26T21:06:07.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:07.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:07.436 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:07.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:07.520+0000 7f739b89ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:07.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:07.521+0000 7f739b89ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:07.534 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:07.535 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:07Z" 2024-05-26T21:06:07.539 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:07.540 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:06:12.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:12.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:12.542 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:12.601 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:12.601 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:12.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:12.633+0000 7fb96c5a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:12.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:12.634+0000 7fb96c5a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:12.647 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:12.648 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:12.649 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T21:06:12.650 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:12.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:12Z" 2024-05-26T21:06:12.652 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:12.653 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.395458_627.1 2024-05-26T21:06:17.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:17.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:17.654 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:17.762 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:17.762 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:17.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:17.797+0000 7f8c7a630a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:17.798+0000 7f8c7a630a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:17.813 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:17.815 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:17.816 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:17.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:17.818 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:17Z" 2024-05-26T21:06:17.818 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:17.819 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.379895_623.1 2024-05-26T21:06:22.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:22.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:22.820 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:22.878 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:22.878 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:22.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:22.914+0000 7f2102f5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:22.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:22.915+0000 7f2102f5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:22.929 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:22.930 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:22.931 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:06:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:22.933 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:22Z" 2024-05-26T21:06:22.934 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:22.935 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:06:27.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:27.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:27.936 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:28.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:28.033+0000 7fe980003a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:28.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:28.042+0000 7fe980003a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:28.137 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:28.137 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:28.138 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:28.139 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:28.140 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:28.141 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:28.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:28.143 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:28Z" 2024-05-26T21:06:28.143 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:28.145 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.379895_623.1 2024-05-26T21:06:33.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:33.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:33.146 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:33.246 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:33.246 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:33.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:33.300+0000 7fa4c5600a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:33.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:33.301+0000 7fa4c5600a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:33.315 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:33.315 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:33.315 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:33.315 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:33.316 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:33.317 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:33.318 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:33.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:33Z" 2024-05-26T21:06:33.320 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:33.321 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:06:38.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:38.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:38.323 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:38.400 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:38.400 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:38.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:38.436+0000 7f63462aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:38.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:38.437+0000 7f63462aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:38.451 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:38.452 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:38.453 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:38.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:38Z" 2024-05-26T21:06:38.455 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:38.456 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:06:43.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:43.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:43.457 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:43.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:43.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:43.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:43.540+0000 7f5c07089a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:43.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:43.541+0000 7f5c07089a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.526780_637.1", 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.526780Z", 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:43Z" 2024-05-26T21:06:43.560 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:43.561 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.526780_637.1 2024-05-26T21:06:48.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:48.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:48.563 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:48.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:48.654+0000 7f10bd5d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:48.655+0000 7f10bd5d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:48.670 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:48.670 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:48.670 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:48.670 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:48.670 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:48.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:48.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:48.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:48.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:48Z" 2024-05-26T21:06:48.675 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:48.676 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:06:53.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:53.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:53.678 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:53.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:53.765+0000 7f83db3d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:53.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:53.765+0000 7f83db3d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:53Z" 2024-05-26T21:06:53.784 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:53.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:06:58.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:58.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:06:58.787 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:06:58.846 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:06:58.846 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:06:58.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:58.882+0000 7f1de3ff0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:58.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:06:58.883+0000 7f1de3ff0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:06:58.899 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:06:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:06:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:06:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:06:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:06:58Z" 2024-05-26T21:06:58.904 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:06:58.905 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:07:03.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:03.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:03.907 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:04.017+0000 7f35d955ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:04.018+0000 7f35d955ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:04.033 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:04.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:04.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:04.037 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:04.038 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:04.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:04.038 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:04Z" 2024-05-26T21:07:04.038 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:04.039 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:07:09.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:09.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:09.041 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:09.112 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:09.113 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:09.147+0000 7fc78eaaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:09.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:09.148+0000 7fc78eaaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:09.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:09.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.167 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:09.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:09.168 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:09.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:09.168 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:09Z" 2024-05-26T21:07:09.168 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:09.169 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:07:14.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:14.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:14.170 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:14.232 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:14.268+0000 7f8c1141aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:14.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:14.269+0000 7f8c1141aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.460055_632.1", 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.460055Z", 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:14Z" 2024-05-26T21:07:14.293 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:14.294 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.460055_632.1 2024-05-26T21:07:19.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:19.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:19.296 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:19.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:19.447+0000 7fd2b3a74a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:19.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:19.448+0000 7fd2b3a74a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:19.461 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:19.461 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:19.461 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:19.461 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:19.462 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:19.463 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:07:19.464 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:19Z" 2024-05-26T21:07:19.466 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:19.467 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:07:24.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:24.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:24.469 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:24.521 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:24.522 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:24.553+0000 7fa99e636a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:24.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:24.554+0000 7fa99e636a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:24.567 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:24.568 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:24.569 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:24.571 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:24.572 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:24.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:24.573 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:24Z" 2024-05-26T21:07:24.573 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:24.574 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:07:29.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:29.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:29.575 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:29.631 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:29.632 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:29.667+0000 7fe25b949a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:29.668+0000 7fe25b949a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.682 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:29.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:29.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:07:29.685 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:29.686 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:29Z" 2024-05-26T21:07:29.687 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:29.688 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:07:34.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:34.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:34.690 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:34.789+0000 7f14cc85ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:34.790+0000 7f14cc85ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:34.804 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:34.805 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:34.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:34.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:34.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:34Z" 2024-05-26T21:07:34.809 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:34.810 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:07:39.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:39.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:39.811 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:39.870 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:39.870 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:39.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:39.906+0000 7fbebbaeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:39.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:39.907+0000 7fbebbaeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:39.921 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:39.922 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:39.923 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:07:39.924 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:39.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:39Z" 2024-05-26T21:07:39.926 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:39.927 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:07:44.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:44.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:44.929 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:44.987 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:44.987 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:45.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:45.024+0000 7f7b1a67fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:45.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:45.025+0000 7f7b1a67fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:45.040 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.362486_620.1", 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.362486Z", 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:45Z" 2024-05-26T21:07:45.046 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:45.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.362486_620.1 2024-05-26T21:07:50.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:50.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:50.048 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:50.195+0000 7f0985592a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:50.196+0000 7f0985592a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:50.210 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:50.210 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:50.210 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:50.210 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:50.210 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:50.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.309042_617.1", 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.309042Z", 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:50.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:50.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:50Z" 2024-05-26T21:07:50.215 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:50.216 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.309042_617.1 2024-05-26T21:07:55.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:55.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:07:55.217 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:07:55.268 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:07:55.269 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:07:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:55.303+0000 7fa441af8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:07:55.304+0000 7fa441af8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:07:55.317 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:07:55.317 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:07:55.317 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:07:55.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:55.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:55.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:07:55.321 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:55.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:55.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:07:55.322 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:55.323 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:07:55.324 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.325 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:07:55.326 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:07:55.326 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:07:55.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:07:55.326 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:07:55Z" 2024-05-26T21:07:55.326 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:07:55.327 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:08:00.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:00.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:00.329 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:00.384 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:00.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:00.417+0000 7fc708504a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:00.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:00.418+0000 7fc708504a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:00.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:00.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:00Z" 2024-05-26T21:08:00.437 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:00.438 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:08:05.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:05.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:05.440 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:05.497 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:05.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:05.532+0000 7f1bced98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:05.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:05.533+0000 7f1bced98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:05.548 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:05.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:05.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.463112_634.1", 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.463112Z", 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:05.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:05.552 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:05Z" 2024-05-26T21:08:05.553 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:05.554 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.463112_634.1 2024-05-26T21:08:10.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:10.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:10.555 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:10.614 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:10.653+0000 7f7d4cba7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:10.656+0000 7f7d4cba7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:10.673 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:10.674 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:10.675 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.299030_616.1", 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.299030Z", 2024-05-26T21:08:10.676 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:10Z" 2024-05-26T21:08:10.678 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:10.679 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.299030_616.1 2024-05-26T21:08:15.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:15.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:15.682 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:15.778+0000 7fd183c71a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:15.779+0000 7fd183c71a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:15.794 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:15.795 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:15.796 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:15.797 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.379895_623.1", 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:15.798 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.379895Z", 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:15.799 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:15.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:15Z" 2024-05-26T21:08:15.801 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:15.802 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.379895_623.1 2024-05-26T21:08:20.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:20.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:20.804 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:20.984+0000 7efce0be5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:20.987+0000 7efce0be5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.015 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:21.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:21.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:20Z" 2024-05-26T21:08:21.020 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:21.021 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:08:26.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:26.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:26.022 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:26.073 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:26.074 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:26.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:26.106+0000 7f9c8a700a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:26.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:26.107+0000 7f9c8a700a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:26.121 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:26Z" 2024-05-26T21:08:26.127 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:26.128 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:08:31.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:31.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:31.129 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:31.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:31.214+0000 7ff7bf61ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:31.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:31.215+0000 7ff7bf61ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:31.228 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:31.228 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:31.228 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:31.228 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:31.229 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:31.229 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:31.229 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:31.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.229 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:31.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:31.230 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:31.230 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:31.230 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:31.230 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:31.231 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:31.231 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:31.231 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:31.231 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:31.231 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:31.232 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.370262_622.1", 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.370262Z", 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:31.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:31.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:31Z" 2024-05-26T21:08:31.235 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:31.236 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.370262_622.1 2024-05-26T21:08:36.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:36.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:36.237 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:36.293 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:36.293 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:36.330+0000 7f9f94e3da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:36.331+0000 7f9f94e3da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:36.346 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:36.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.398810_629.1", 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.398810Z", 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:36.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:36.350 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:36Z" 2024-05-26T21:08:36.351 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:36.352 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.398810_629.1 2024-05-26T21:08:41.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:41.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:41.354 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:41.406 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:41.407 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:41.439+0000 7f806645ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:41.440+0000 7f806645ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:41.454 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.311444_618.1", 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.311444Z", 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:41.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:41.457 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:41Z" 2024-05-26T21:08:41.458 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:41.459 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.311444_618.1 2024-05-26T21:08:46.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:46.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:46.460 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:46.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:46.555+0000 7f176d9d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:46.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:46.556+0000 7f176d9d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.360000_619.1", 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.360000Z", 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:46.577 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:46.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:46.579 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:46Z" 2024-05-26T21:08:46.579 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:46.580 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.360000_619.1 2024-05-26T21:08:51.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:51.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:51.581 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:51.639 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:51.639 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:51.676+0000 7f1fd755ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:51.677+0000 7f1fd755ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:51.691 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:51.691 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:51.692 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:51.693 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.395458_627.1", 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.395458Z", 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:51.695 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:51Z" 2024-05-26T21:08:51.696 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:51.697 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.395458_627.1 2024-05-26T21:08:56.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:56.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:08:56.699 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:08:56.754 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:08:56.754 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:08:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:56.789+0000 7fd5249d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:08:56.790+0000 7fd5249d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:08:56.804 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:56.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:56.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:56.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:56.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:08:56Z" 2024-05-26T21:08:56.809 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:08:56.810 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:09:01.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:01.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:01.811 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:01.876 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:01.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:01.910+0000 7f9063131a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:01.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:01.911+0000 7f9063131a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:09:01.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:01.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.367915_621.1", 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.367915Z", 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:01.928 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.929 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:09:01.930 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:09:01.930 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:09:01.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:01.930 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:09:01Z" 2024-05-26T21:09:01.930 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:01.931 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.367915_621.1 2024-05-26T21:09:06.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:06.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:06.933 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:06.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:06.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:07.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:07.030+0000 7f7542aeba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:07.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:07.031+0000 7f7542aeba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:07.044 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:07.044 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:07.045 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:07.046 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296898_615.1", 2024-05-26T21:09:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296898Z", 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:07.048 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:09:07Z" 2024-05-26T21:09:07.049 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:07.050 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296898_615.1 2024-05-26T21:09:12.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:12.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:12.051 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:12.110 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:12.111 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:12.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:12.148+0000 7fd0a2392a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:12.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:12.149+0000 7fd0a2392a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:12.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:12.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:12.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.296023_614.1", 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:12.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.296023Z", 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:09:12.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:09:12Z" 2024-05-26T21:09:12.169 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:12.170 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757079.172867_790.1 target=1_1716756067.296023_614.1 2024-05-26T21:09:17.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2024-05-26T21:09:17.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.173 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "c76995de-b7a3-48fd-992e-bd9377418b2b", 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "remove_zg", 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "remove_zg", 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": false, 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T21:09:17.250 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "", 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [], 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [] 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T21:09:17.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.252 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T21:09:17.252 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.252 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.252 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:17.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.253 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.302 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.303 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:17.330+0000 7fcff4453a80 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-05-26T21:09:17.333 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:17.333 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "9aedc5ee-92fd-43c2-86a0-bfb9f4313cdd", 2024-05-26T21:09:17.333 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "remove", 2024-05-26T21:09:17.333 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "remove.rgw.meta:root", 2024-05-26T21:09:17.333 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "remove.rgw.control", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "remove.rgw.log:gc", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "remove.rgw.log:lc", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "remove.rgw.log", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-05-26T21:09:17.334 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "remove.rgw.otp", 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "remove.rgw.buckets.index", 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-05-26T21:09:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "remove.rgw.buckets.data" 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0, 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "inline_data": true 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "remove.rgw.log:notif" 2024-05-26T21:09:17.336 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:17.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.338 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.386 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.387 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 5, 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.438 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "c76995de-b7a3-48fd-992e-bd9377418b2b", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "remove_zg", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "remove_zg", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": false, 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "9aedc5ee-92fd-43c2-86a0-bfb9f4313cdd", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "9aedc5ee-92fd-43c2-86a0-bfb9f4313cdd", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "remove", 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-05-26T21:09:17.439 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": false, 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.440 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T21:09:17.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T21:09:17.442 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:17.443 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T21:09:17.444 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:17.445 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.446 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.454 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T21:09:17.455 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T21:09:17.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "9aedc5ee-92fd-43c2-86a0-bfb9f4313cdd", 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3399653301 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.457 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.458 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:17.459 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:17.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:17.461 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:17.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.462 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.509 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.509 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:17.531+0000 7f6279909a80 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-05-26T21:09:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:17.532+0000 7f6279909a80 0 Removed zone from zonegroup remove_zg 2024-05-26T21:09:17.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.540 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.596 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.596 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:09:17.638+0000 7fd750bf8a80 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-05-26T21:09:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-26T21:09:17.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.645 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:17.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:09:17.737 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:09:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:18.026 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:09:18.026 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:18.026 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 6, 2024-05-26T21:09:18.026 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T21:09:18.026 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:18.027 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T21:09:18.028 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:18.029 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.030 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:18.031 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T21:09:18.032 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T21:09:18.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:18.034 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T21:09:18.036 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T21:09:18.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.036 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T21:09:18.036 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:09:18.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T21:09:18.070 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T21:09:18.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:18.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:18.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T21:09:18.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T21:09:18.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:09:18.073 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:09:18.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-05-26T21:09:18.074 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-49 2024-05-26T21:09:18.526 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-50 2024-05-26T21:09:18.571 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-51 2024-05-26T21:09:18.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-05-26T21:09:18.622 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-52 2024-05-26T21:09:18.649 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-53 2024-05-26T21:09:18.688 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-54 2024-05-26T21:09:20.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-05-26T21:09:20.759 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-55 2024-05-26T21:09:20.777 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-56 2024-05-26T21:09:20.809 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-57 2024-05-26T21:09:20.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:20.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:20.842 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:20.895 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:20.895 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:09:20.940 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:09:20.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:20.940 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.829504_863.1", 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:09:20.829504Z" 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:20.941 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:09:20.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:09:20.942 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:09:20.942 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:09:20.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:20.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:20.943 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:20.999 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:20.999 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:21.032 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:21.032+0000 7fe4388b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:21.032 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:21.033+0000 7fe4388b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:21.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:21.047 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:21.048 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.049 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:21.050 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:21.050 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:21.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:21.050 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:21Z" 2024-05-26T21:09:21.050 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:21.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:26.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:26.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:26.052 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:26.108 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:26.109 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:26.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:26.141+0000 7fcc2bb89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:26.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:26.141+0000 7fcc2bb89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:26.153 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:26Z" 2024-05-26T21:09:26.159 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:26.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:31.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:31.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:31.162 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:31.219 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:31.250 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:31.252+0000 7f6f83eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:31.251 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:31.253+0000 7f6f83eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:31.264 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:31.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:31.269 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:31Z" 2024-05-26T21:09:31.270 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:31.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:36.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:36.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:36.274 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:36.331 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:36.331 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:36.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:36.364+0000 7f09cc17da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:36.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:36.365+0000 7f09cc17da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:36.376 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:36.376 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:36.376 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:36.376 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:36.377 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:36.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:36.379 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:36.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:36Z" 2024-05-26T21:09:36.381 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:36.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:41.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:41.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:41.384 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:41.440 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:41.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:41.473 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:41.473+0000 7f95360c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:41.474 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:41.474+0000 7f95360c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:41.488 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:41.488 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:41.489 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:41.490 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:41.491 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:41.492 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:41Z" 2024-05-26T21:09:41.493 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:41.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:46.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:46.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:46.495 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:46.552 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:46.553 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:46.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:46.583+0000 7f684a7b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:46.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:46.584+0000 7f684a7b4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:46.597 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:46Z" 2024-05-26T21:09:46.602 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:46.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:51.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:51.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:51.605 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:51.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:51.662 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:51.692 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:51.693+0000 7f9b11840a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:51.693 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:51.694+0000 7f9b11840a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:51.706 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:51.706 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:51.706 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:51.706 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:51.706 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:51.707 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:51.708 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:51.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:51.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:51Z" 2024-05-26T21:09:51.711 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:51.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:09:56.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:56.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:09:56.713 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:09:56.771 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:09:56.771 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:09:56.802 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:56.803+0000 7f9f0d4a9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:56.803 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:09:56.804+0000 7f9f0d4a9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:09:56.816 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:09:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:09:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:09:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:09:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:09:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:09:56Z" 2024-05-26T21:09:56.822 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:09:56.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:01.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:01.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:01.824 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:01.881 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:01.882 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:01.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:01.913+0000 7f7aa54f9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:01.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:01.914+0000 7f7aa54f9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:01.927 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:01.927 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:01.928 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:01.929 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:01.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:01.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:01.932 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:01Z" 2024-05-26T21:10:01.933 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:01.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:06.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:06.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:06.936 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:06.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:06.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:07.023 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:07.024+0000 7f3d0de4da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:07.024 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:07.025+0000 7f3d0de4da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:07.037 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:07.042 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:07.043 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:07.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:07.043 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:07Z" 2024-05-26T21:10:07.043 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:07.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:12.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:12.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:12.045 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:12.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:12.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:12.133 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:12.133+0000 7f993d983a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:12.134 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:12.134+0000 7f993d983a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:12.146 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:12.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:12Z" 2024-05-26T21:10:12.151 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:12.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:17.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:17.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:17.153 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:17.210 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:17.210 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:17.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:17.244+0000 7f24b8b8ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:17.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:17.245+0000 7f24b8b8ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:17.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:17Z" 2024-05-26T21:10:17.262 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:17.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:22.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:22.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:22.265 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:22.322 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:22.323 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:22.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:22.355+0000 7fe8ea913a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:22.354 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:22.356+0000 7fe8ea913a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:22.367 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:22.367 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:22Z" 2024-05-26T21:10:22.372 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:22.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:27.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:27.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:27.375 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:27.432 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:27.432 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:27.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:27.464+0000 7f394b0f2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:27.464 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:27.465+0000 7f394b0f2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:27.477 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:27.478 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:27Z" 2024-05-26T21:10:27.482 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:27.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:32.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:32.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:32.484 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:32.541 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:32.541 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:32.574 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:32.575+0000 7f6b12fcca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:32.575 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:32.576+0000 7f6b12fcca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:32.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:32.590 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:32.591 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:32.592 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:32Z" 2024-05-26T21:10:32.593 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:32.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:37.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:37.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:37.595 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:37.652 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:37.653 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:37.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:37.684+0000 7f874d52ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:37.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:37.685+0000 7f874d52ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:37.698 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:37.698 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:37.698 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:37.699 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:37.700 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:37.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:37.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:37.703 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:37Z" 2024-05-26T21:10:37.704 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:37.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:42.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:42.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:42.706 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:42.763 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:42.763 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:42.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:42.795+0000 7ff29c7cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:42.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:42.796+0000 7ff29c7cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:42.809 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:42.809 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:42.809 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:42.809 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:42.809 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:42.810 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:42.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:42Z" 2024-05-26T21:10:42.814 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:42.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:47.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:47.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:47.817 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:47.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:47.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:47.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:47.907+0000 7f5a2f9eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:47.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:47.908+0000 7f5a2f9eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:47.920 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:47.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:47.925 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:47.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:47.926 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:47Z" 2024-05-26T21:10:47.926 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:47.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:52.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:52.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:52.928 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:52.985 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:52.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:53.017 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:53.017+0000 7f882c34ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:53.017 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:53.018+0000 7f882c34ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:53.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:53Z" 2024-05-26T21:10:53.035 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:53.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:10:58.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:58.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:10:58.037 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:10:58.095 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:10:58.095 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:10:58.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:58.128+0000 7f6db6e2ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:58.128 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:10:58.128+0000 7f6db6e2ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:10:58.141 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:10:58.141 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:10:58.141 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:10:58.141 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:10:58.142 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:58.143 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:10:58.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:10:58Z" 2024-05-26T21:10:58.146 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:10:58.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:11:03.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:03.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:03.148 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:03.205 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:03.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:03.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:03.238+0000 7f09107f7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:03.239 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:03.239+0000 7f09107f7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:03.254 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757079.172867_790.1", 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:57:59.172867Z", 2024-05-26T21:11:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:03.256 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:03.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:03.258 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:03Z" 2024-05-26T21:11:03.258 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:03.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757079.172867_790.1 2024-05-26T21:11:08.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:08.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:08.260 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:08.317 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:08.317 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:08.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:08.349+0000 7fee2d4a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:08.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:08.350+0000 7fee2d4a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.505181_811.1", 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.505181Z", 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:08Z" 2024-05-26T21:11:08.368 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:08.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.505181_811.1 2024-05-26T21:11:13.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:13.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:13.371 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:13.429 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:13.429 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:13.463 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:13.464+0000 7f53d351ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:13.464 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:13.465+0000 7f53d351ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:13.477 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:13.477 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:13.477 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:13.477 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:13.477 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:13.478 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:13.479 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:13.480 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:13.480 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:13.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:13.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:13.481 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:13.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:11:13.483 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:13.485 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.486 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:13.487 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:13Z" 2024-05-26T21:11:13.487 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:13.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.589013_824.1 2024-05-26T21:11:18.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:18.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:18.491 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:18.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:18.583+0000 7f52c7905a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:18.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:18.584+0000 7f52c7905a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:18.621 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:18.621 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:18.621 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:18.621 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:18.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:18.623 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.601289_827.1", 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:18.624 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.601289Z", 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:18.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:18Z" 2024-05-26T21:11:18.626 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:18.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.601289_827.1 2024-05-26T21:11:23.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:23.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:23.628 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:23.685 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:23.685 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:23.717 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:23.718+0000 7fb1c4f4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:23.717 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:23.719+0000 7fb1c4f4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:23.730 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:23.731 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:23.732 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:23.733 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:23.734 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:23.735 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:23.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:23.736 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:23Z" 2024-05-26T21:11:23.736 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:23.737 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:11:28.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:28.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:28.738 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:28.793 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:28.793 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:28.824 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:28.825+0000 7fbfe112aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:28.825 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:28.826+0000 7fbfe112aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:28.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:28.840 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:28.841 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:28.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:28Z" 2024-05-26T21:11:28.843 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:28.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:11:33.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:33.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:33.845 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:33.934 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:33.934+0000 7f74102f8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:33.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:33.935+0000 7f74102f8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:33.948 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:33.948 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:33.948 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:33.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:33.950 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:33.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:33.952 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:33Z" 2024-05-26T21:11:33.953 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:33.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:11:38.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:38.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:38.955 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:39.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:39.044+0000 7fdd8a6e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:39.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:39.045+0000 7fdd8a6e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:39.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:11:39.063 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:39.064 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:39.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:39.066 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:39Z" 2024-05-26T21:11:39.066 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:39.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.589013_824.1 2024-05-26T21:11:44.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:44.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:44.068 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:44.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:44.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:44.164 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:44.164+0000 7fb055539a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:44.165 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:44.165+0000 7fb055539a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:44.178 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:44.179 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:44.180 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:44.181 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:44.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:44Z" 2024-05-26T21:11:44.183 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:44.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:11:49.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:49.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:49.185 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:49.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:49.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:49.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:49.275+0000 7f200f3fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:49.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:49.276+0000 7f200f3fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:49Z" 2024-05-26T21:11:49.293 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:49.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.589013_824.1 2024-05-26T21:11:54.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:54.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:54.296 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:54.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:54.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:54.384 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:54.384+0000 7ff7144f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:54.384 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:54.385+0000 7ff7144f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:54.397 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:54.397 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:54.397 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:54.398 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:54.399 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:54.400 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:54.401 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:54Z" 2024-05-26T21:11:54.402 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:54.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:11:59.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:59.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:11:59.404 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:11:59.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:11:59.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:11:59.494 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:59.495+0000 7fdeb03f7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:59.495 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:11:59.496+0000 7fdeb03f7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:11:59.509 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:11:59.509 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:11:59.509 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:59.510 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:59.511 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:59.512 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:11:59.513 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:11:59Z" 2024-05-26T21:11:59.514 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:11:59.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:12:04.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:04.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:04.516 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:04.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:04.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:04.607 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:04.608+0000 7fbcb638ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:04.608 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:04.609+0000 7fbcb638ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:12:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:12:04.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:04.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:04.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:12:04.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:12:04Z" 2024-05-26T21:12:04.626 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:12:04.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.589013_824.1 2024-05-26T21:12:09.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:09.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:09.628 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:09.683 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:09.683 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:09.713 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:09.713+0000 7ffbe4c69a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:09.714+0000 7ffbe4c69a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:12:09.727 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:09.728 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:09.729 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:09.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:12:09Z" 2024-05-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:12:09.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:12:14.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:14.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:14.734 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:14.787 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:14.787 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:14.817 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:14.819+0000 7fdfdc54da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:14.818 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:14.820+0000 7fdfdc54da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:12:14.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757758.633767_828.1", 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:18.633767Z", 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.835 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:12:14.836 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:12:14.836 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:12:14.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:14.836 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:12:14Z" 2024-05-26T21:12:14.836 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:12:14.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716757758.633767_828.1 2024-05-26T21:12:19.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:19.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:19.839 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:19.895 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:19.895 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:19.926 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:19.927+0000 7fdc774d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:19.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:12:19.928+0000 7fdc774d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:19.940 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:12:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:12:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:12:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:12:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757760.829504_863.1", 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:20.829504Z", 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:12:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:12:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:12:19Z" 2024-05-26T21:12:19.946 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:12:19.947 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:12:19.947 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:12:19.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:19.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:19.948 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:20.001 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:20.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:20.033+0000 7f8aa0d33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:20.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:20.034+0000 7f8aa0d33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:20.047 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:20.048 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:20.050 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:20.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:20.052 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:20.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:20.052 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:20Z" 2024-05-26T21:12:20.052 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:20.053 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756387.720498_718.1 2024-05-26T21:12:25.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:25.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:25.055 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:25.113 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:25.113 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:25.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:25.151+0000 7f59abdf5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:25.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:25.153+0000 7f59abdf5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:25.169 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:25.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:25.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:25Z" 2024-05-26T21:12:25.175 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:25.176 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:12:30.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:30.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:30.178 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:30.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:30.321+0000 7f0721f58a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:30.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:30.322+0000 7f0721f58a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:30.335 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:30.335 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:30.335 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:30.335 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:30.335 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:30.336 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:30.339 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:30Z" 2024-05-26T21:12:30.340 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:30.341 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756068.677571_656.1 2024-05-26T21:12:35.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:35.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:35.343 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:35.426+0000 7f4451205a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:35.427+0000 7f4451205a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:35.439 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:35.441 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:35.442 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:35.443 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:35.444 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:35Z" 2024-05-26T21:12:35.445 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:35.446 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.735465_658.1 2024-05-26T21:12:40.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:40.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:40.447 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:40.507 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:40.507 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:40.544+0000 7f0b838e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:40.545+0000 7f0b838e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:40.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:40.562 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:40.563 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:40.564 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:40Z" 2024-05-26T21:12:40.565 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:40.566 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.735465_658.1 2024-05-26T21:12:45.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:45.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:45.567 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:45.627 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:45.627 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:45.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:45.663+0000 7febcbb68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:45.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:45.664+0000 7febcbb68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:45Z" 2024-05-26T21:12:45.684 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:45.684 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:12:50.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:50.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:50.686 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:50.770+0000 7fdc98628a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:50.771+0000 7fdc98628a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:50.785 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:50.785 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:50.785 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:50.785 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:50.786 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:50.787 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:12:50.788 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:50Z" 2024-05-26T21:12:50.790 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:50.791 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.632586_650.1 2024-05-26T21:12:55.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:55.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:12:55.793 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:12:55.849 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:12:55.849 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:12:55.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:55.885+0000 7feed609ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:55.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:12:55.886+0000 7feed609ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:55.903 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:12:55.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:12:55Z" 2024-05-26T21:12:55.905 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:12:55.906 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:13:00.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:00.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:00.907 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:01.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:01.001+0000 7f97aaf90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:01.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:01.002+0000 7f97aaf90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:01Z" 2024-05-26T21:13:01.020 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:01.021 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756387.720498_718.1 2024-05-26T21:13:06.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:06.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:06.023 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:06.118+0000 7ffbf4283a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:06.119+0000 7ffbf4283a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:06.133 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:06.133 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:06.133 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:06.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:06.137 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:06Z" 2024-05-26T21:13:06.138 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:06.139 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756068.677571_656.1 2024-05-26T21:13:11.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:11.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:11.141 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:11.232 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:11.233 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:11.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:11.353+0000 7f749a29ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:11.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:11.354+0000 7f749a29ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:11.368 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:11.370 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.372 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:11.373 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:11.373 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:11.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:11.373 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:11Z" 2024-05-26T21:13:11.373 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:11.374 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:13:16.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:16.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:16.375 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:16.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:16.463+0000 7f9deea3fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:16.464+0000 7f9deea3fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:16.477 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:16.477 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:16.478 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:16.479 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:16.480 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:16.481 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:16Z" 2024-05-26T21:13:16.482 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:16.483 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.792493_659.1 2024-05-26T21:13:21.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:21.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:21.485 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:21.544 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:21.544 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:21.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:21.580+0000 7f9f68922a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:21.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:21.581+0000 7f9f68922a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:21Z" 2024-05-26T21:13:21.601 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:21.602 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.731394_657.1 2024-05-26T21:13:26.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:26.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:26.604 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:26.663 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:26.663 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:26.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:26.699+0000 7f88b6657a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:26.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:26.700+0000 7f88b6657a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:26.715 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:26.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:26Z" 2024-05-26T21:13:26.720 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:26.721 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:13:31.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:31.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:31.723 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:31.830+0000 7f1708326a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:31.831+0000 7f1708326a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:31.844 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:31.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:31.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:31.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:31Z" 2024-05-26T21:13:31.850 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:31.851 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.792493_659.1 2024-05-26T21:13:36.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:36.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:36.852 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:36.905 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:36.905 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:36.937+0000 7ff4ae252a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:36.938+0000 7ff4ae252a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:36.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:36.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:36.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:36.959 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:36Z" 2024-05-26T21:13:36.960 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:36.961 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:13:41.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:41.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:41.965 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:42.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:42.060+0000 7f2b8ab18a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:42.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:42.061+0000 7f2b8ab18a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:42.075 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:42.075 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:42.075 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:42.075 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:42.076 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:42.077 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:42.078 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:42.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:42Z" 2024-05-26T21:13:42.080 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:42.081 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:13:47.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:47.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:47.083 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:47.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:47.180+0000 7f5a3006aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:47.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:47.181+0000 7f5a3006aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:47.197 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:47.198 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:47.199 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:47.200 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:47.201 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:47Z" 2024-05-26T21:13:47.202 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:47.203 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:13:52.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:52.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:52.205 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:52.270 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:52.270 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:52.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:52.303+0000 7f6a37da1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:52.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:52.304+0000 7f6a37da1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:52.318 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:52.318 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:52.318 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:52.318 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:52.318 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:52.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:52.320 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:52.321 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:52.322 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:52Z" 2024-05-26T21:13:52.323 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:52.324 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:13:57.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:57.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:13:57.325 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:13:57.380 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:13:57.380 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:13:57.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:57.414+0000 7f8b49830a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:57.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:13:57.415+0000 7f8b49830a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:57.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:13:57.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:57.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:57.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:13:57.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:13:57.434 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:13:57Z" 2024-05-26T21:13:57.435 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:13:57.436 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.572561_648.1 2024-05-26T21:14:02.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:02.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:02.438 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:02.497 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:02.497 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:02.533+0000 7f8fbc11ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:02.534+0000 7f8fbc11ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:02.548 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:02.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:02.550 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:02.551 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:02.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:02.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:02Z" 2024-05-26T21:14:02.555 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:02.556 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:14:07.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:07.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:07.558 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:07.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:07.654+0000 7f41a1c16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:07.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:07.664+0000 7f41a1c16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:07.755 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:07Z" 2024-05-26T21:14:07.761 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:07.762 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.632586_650.1 2024-05-26T21:14:12.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:12.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:12.764 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:12.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:12.848+0000 7f419d2b9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:12.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:12.849+0000 7f419d2b9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:12.862 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:12.862 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:12.862 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:12.862 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:12.863 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:12.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:14:12.865 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:12.866 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:12Z" 2024-05-26T21:14:12.867 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:12.869 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.635928_652.1 2024-05-26T21:14:17.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:17.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:17.870 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:17.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:17.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:17.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:17.965+0000 7fda9b53ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:17.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:17.966+0000 7fda9b53ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:14:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:17Z" 2024-05-26T21:14:17.987 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:17.989 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.633621_651.1 2024-05-26T21:14:22.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:22.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:22.990 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:23.050 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:23.050 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:23.086+0000 7f0b6c4bda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:23.087+0000 7f0b6c4bda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:23.102 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:23.102 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:23.103 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:23.104 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:23.105 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:23.106 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:23.107 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:23.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:23.108 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:23Z" 2024-05-26T21:14:23.108 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:23.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:14:28.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:28.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:28.111 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:28.207+0000 7f5d2dd97a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:28.208+0000 7f5d2dd97a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:28.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:28.224 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:28.225 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:28.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:28.227 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:28Z" 2024-05-26T21:14:28.228 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:28.229 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756068.677571_656.1 2024-05-26T21:14:33.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:33.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:33.233 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:33.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:33.370+0000 7f5525feba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:33.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:33.371+0000 7f5525feba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:33.384 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:33.384 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:33.384 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:33.384 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:33.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:33.386 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:33.387 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:33.388 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:33Z" 2024-05-26T21:14:33.390 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:33.390 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.635928_652.1 2024-05-26T21:14:38.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:38.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:38.392 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:38.444 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:38.444 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:38.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:38.477+0000 7f323e23da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:38.479+0000 7f323e23da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:38.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:38.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.747363_723.1", 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.747363Z", 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:38.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:38.496 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:38.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:38.497 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:38Z" 2024-05-26T21:14:38.497 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:38.498 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756387.747363_723.1 2024-05-26T21:14:43.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:43.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:43.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:43.552 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:43.552 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:43.585+0000 7fa12951ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:43.586+0000 7fa12951ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:43Z" 2024-05-26T21:14:43.606 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:43.609 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:14:48.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:48.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:48.610 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:48.708+0000 7f6bfcb10a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:48.709+0000 7f6bfcb10a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:48.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:48.726 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:48.727 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:48.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:48.729 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:48.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:48.729 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:48Z" 2024-05-26T21:14:48.729 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:48.730 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:14:53.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:53.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:53.732 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:53.828+0000 7f2043ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:53.829+0000 7f2043ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:53.843 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:53Z" 2024-05-26T21:14:53.848 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:53.850 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:14:58.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:58.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:14:58.851 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:14:58.910 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:14:58.910 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:14:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:58.947+0000 7f7056119a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:14:58.948+0000 7f7056119a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:14:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:14:58.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:14:58.966 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:14:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:14:58Z" 2024-05-26T21:14:58.969 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:14:58.970 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:15:03.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:03.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:03.971 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:04.144+0000 7f6d394c7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:04.146+0000 7f6d394c7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:04.160 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:04.160 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:04.160 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:04.160 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:04.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:04.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:04.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:04Z" 2024-05-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:04.166 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:15:09.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:09.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:09.167 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:09.220 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:09.220 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:09.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:09.252+0000 7f98aae5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:09.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:09.253+0000 7f98aae5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:09.266 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:09.267 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:09.268 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:09.269 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:09.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:09.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:09.272 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:09Z" 2024-05-26T21:15:09.272 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:09.273 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.648499_654.1 2024-05-26T21:15:14.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:14.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:14.274 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:14.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:14.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:14.371+0000 7f0034e38a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:14.372+0000 7f0034e38a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:14.387 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756705.971680_773.1", 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:51:45.971680Z", 2024-05-26T21:15:14.390 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:14.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:14Z" 2024-05-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:14.393 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756705.971680_773.1 2024-05-26T21:15:19.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:19.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:19.395 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:19.491+0000 7fc170928a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:19.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:19.492+0000 7fc170928a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:19.511 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:19.511 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:19.511 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:19.511 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:19.512 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:19.513 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:15:19.514 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:19.515 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:19Z" 2024-05-26T21:15:19.516 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:19.517 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.645969_653.1 2024-05-26T21:15:24.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:24.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:24.519 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:24.622 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:24.622 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:24.677+0000 7f3e62f3ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:24.678+0000 7f3e62f3ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:24.691 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:24.691 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:24.691 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:24.691 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:24.691 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:24.692 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:24.693 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:15:24.694 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:24.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:24Z" 2024-05-26T21:15:24.696 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:24.697 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.813012_663.1 2024-05-26T21:15:29.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:29.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:29.699 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:29.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:29.780+0000 7f8549d6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:29.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:29.781+0000 7f8549d6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:29.795 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:29.796 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:29.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:29.798 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:29.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:29Z" 2024-05-26T21:15:29.800 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:29.801 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:15:34.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:34.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:34.803 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:34.856 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:34.856 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:34.890+0000 7fb505ce5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:34.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:34.891+0000 7fb505ce5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:34.905 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:34.906 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:34.907 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:34.908 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:34.909 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:34Z" 2024-05-26T21:15:34.910 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:34.911 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:15:39.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:39.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:39.913 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:39.999+0000 7ff87ffa5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:40.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:40.000+0000 7ff87ffa5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:40.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:40.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:40.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:40Z" 2024-05-26T21:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:40.020 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.572561_648.1 2024-05-26T21:15:45.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:45.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:45.021 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:45.080 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:45.080 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:45.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:45.116+0000 7ffab436ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:45.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:45.117+0000 7ffab436ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:45.132 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:45Z" 2024-05-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:45.139 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.731394_657.1 2024-05-26T21:15:50.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:50.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:50.140 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:50.201 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:50.201 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:50.361+0000 7f106d963a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:50.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:50.363+0000 7f106d963a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:50.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:50.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:50.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:50.381 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:50.382 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:50.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:50.383 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:50Z" 2024-05-26T21:15:50.383 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:50.384 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:15:55.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:55.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:15:55.385 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:15:55.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:15:55.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:15:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:55.469+0000 7f58f8377a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:15:55.470+0000 7f58f8377a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:15:55.482 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:15:55.482 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:15:55.482 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:55.484 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:15:55.485 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:15:55.487 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:15:55Z" 2024-05-26T21:15:55.488 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:15:55.489 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:16:00.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:00.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:00.491 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:00.545 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:00.545 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:00.579+0000 7ff7d0d54a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:00.580+0000 7ff7d0d54a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:00.593 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:00.593 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:00.593 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:00.593 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:00.594 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:00.595 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:00.596 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:00.597 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:00Z" 2024-05-26T21:16:00.598 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:00.599 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.648499_654.1 2024-05-26T21:16:05.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:05.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:05.600 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:05.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:05.659 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:05.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:05.695+0000 7f838958ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:05.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:05.696+0000 7f838958ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:05.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:05.712 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:05.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.715 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:05.716 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:05.716 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:05.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:05.716 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:05Z" 2024-05-26T21:16:05.716 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:05.717 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756068.677571_656.1 2024-05-26T21:16:10.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:10.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:10.719 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:10.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:10.813+0000 7f5e20d4ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:10.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:10.814+0000 7f5e20d4ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.829 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:10.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:10.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:10.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:10Z" 2024-05-26T21:16:10.834 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:10.835 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:16:15.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:15.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:15.836 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:15.887 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:15.887 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:15.920+0000 7fd1e3253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:15.921+0000 7fd1e3253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:15.935 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:15.935 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:15.935 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:15.935 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:15.935 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:15.936 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:15.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:15.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:15.939 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:15Z" 2024-05-26T21:16:15.940 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:15.941 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:16:20.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:20.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:20.943 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:20.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:20.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:21.034+0000 7f196ef9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:21.035+0000 7f196ef9da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:21.050 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:21.050 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:21.050 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:21.050 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:21.051 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:21.052 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:16:21.053 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:21.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:21Z" 2024-05-26T21:16:21.055 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:21.056 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:16:26.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:26.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:26.057 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:26.116 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:26.116 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:26.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:26.152+0000 7fa6c2605a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:26.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:26.153+0000 7fa6c2605a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:26.168 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:26.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:26Z" 2024-05-26T21:16:26.174 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:26.175 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.731394_657.1 2024-05-26T21:16:31.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:31.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:31.176 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:31.249 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:31.282+0000 7f65d759fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:31.283+0000 7f65d759fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:31.296 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:31.297 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:31.298 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:16:31.299 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:31.300 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:31Z" 2024-05-26T21:16:31.301 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:31.302 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:16:36.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:36.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:36.304 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:36.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:36.388+0000 7fa72ad1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:36.389+0000 7fa72ad1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:36.402 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:36Z" 2024-05-26T21:16:36.409 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:36.411 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.632586_650.1 2024-05-26T21:16:41.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:41.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:41.412 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:41.470 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:41.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:41.506+0000 7efe6c879a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:41.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:41.507+0000 7efe6c879a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:41.522 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:41.523 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:41.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:41Z" 2024-05-26T21:16:41.527 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:41.528 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756069.792493_659.1 2024-05-26T21:16:46.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:46.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:46.529 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:46.588 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:46.588 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:46.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:46.624+0000 7f4caa67aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:46.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:46.625+0000 7f4caa67aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:46Z" 2024-05-26T21:16:46.644 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:46.645 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:16:51.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:51.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:51.647 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:51.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:51.856+0000 7f5405719a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:51.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:51.858+0000 7f5405719a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:51.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:51Z" 2024-05-26T21:16:51.877 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:51.878 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.633621_651.1 2024-05-26T21:16:56.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:56.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:16:56.880 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:16:56.930 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:16:56.930 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:16:56.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:56.962+0000 7f05b40a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:56.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:16:56.963+0000 7f05b40a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:16:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:16:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:16:56Z" 2024-05-26T21:16:56.982 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:16:56.983 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.570363_647.1 2024-05-26T21:17:01.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:01.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:01.985 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:02.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:02.081+0000 7f33e5c44a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:02.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:02.082+0000 7f33e5c44a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:02.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:02.099 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:02.100 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:02.101 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:02.101 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:02.101 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:17:02.101 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:02.102 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:02.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:02.102 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:17:02.102 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:02.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:02.103 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:02.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:02.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:02.106 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:02Z" 2024-05-26T21:17:02.106 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:02.108 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.629964_649.1 2024-05-26T21:17:07.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:07.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:07.109 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:07.168 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:07.168 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:07.210+0000 7fbc60b6ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:07.211+0000 7fbc60b6ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:07.225 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:07Z" 2024-05-26T21:17:07.231 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:07.232 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:17:12.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:12.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:12.233 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:12.339 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:12.339 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:12.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:12.431+0000 7fa6ecd75a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:12.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:12.432+0000 7fa6ecd75a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:12.445 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:12.450 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:12.451 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:12.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:12.451 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:12Z" 2024-05-26T21:17:12.451 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:12.452 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.572561_648.1 2024-05-26T21:17:17.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:17.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:17.453 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:17.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:17.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:17.535+0000 7f15723a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:17.536+0000 7f15723a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:17.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:17.551 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:17.552 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756705.907405_765.1", 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:51:45.907405Z", 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:17.553 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:17Z" 2024-05-26T21:17:17.554 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:17.555 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756705.907405_765.1 2024-05-26T21:17:22.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:22.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:22.557 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:22.654 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:22.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:22.648+0000 7f929f059a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:22.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:22.649+0000 7f929f059a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:22.663 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:22.663 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:22.663 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:22.663 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:22.664 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:17:22.667 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:22.668 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:22Z" 2024-05-26T21:17:22.671 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:22.672 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716757760.829504_863.1 target=1_1716756067.569434_646.1 2024-05-26T21:17:27.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-05-26T21:17:27.674 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-58 2024-05-26T21:17:27.697 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-59 2024-05-26T21:17:27.734 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-60 2024-05-26T21:17:28.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:28.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:28.087 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:28.137 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:28.137 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:17:28.181 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:17:28.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:28.181 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:28.181 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758247.756860_875.1", 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:17:27.756860Z" 2024-05-26T21:17:28.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:28.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:28.183 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:28.183 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:17:28.183 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:28.183 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:17:28.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:17:28.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:28.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:28.184 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:28.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:28.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:28.273 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:17:28.273+0000 7f9970ad9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:28.274 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:17:28.274+0000 7f9970ad9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:17:28.290 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:17:28.291 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:28.292 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:28.293 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716757760.829504_863.1", 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:28.295 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:09:20.829504Z", 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:17:28.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:17:28Z" 2024-05-26T21:17:28.299 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:17:28.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716757760.829504_863.1 2024-05-26T21:17:29.515 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T21:17:29.501+0000 7f30939af700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-05-26T21:17:30.673 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:17:30.673+0000 7f2530ee9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:17:33.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:33.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:33.301 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:33.357 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:33.358 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:33.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:17:33.390+0000 7f86366f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:33.390 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:17:33.391+0000 7f86366f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:17:33.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:17:33.406 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:33.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:33.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:33.409 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716758247.756860_875.1", 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:17:27.756860Z", 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:33.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:17:33.411 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:17:33Z" 2024-05-26T21:17:33.412 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:17:33.413 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:17:33.414 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:17:33.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:33.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:33.414 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:33.511+0000 7f784f526a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:33.513+0000 7f784f526a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:33.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:33.529 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:33.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:33.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:33.532 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:33Z" 2024-05-26T21:17:33.533 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:33.534 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:17:38.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:38.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:38.536 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:38.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:38.631+0000 7fd92198fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:38.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:38.632+0000 7fd92198fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:38Z" 2024-05-26T21:17:38.651 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:38.652 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.632586_650.1 2024-05-26T21:17:39.505 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:17:39.505+0000 7f252bedf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:17:39.505 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:17:39.505+0000 7f2527ed7700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:17:43.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:43.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:43.654 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:43.758 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:43.758 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:43.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:43.801+0000 7f082b7afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:43.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:43.802+0000 7f082b7afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:43Z" 2024-05-26T21:17:43.821 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:43.822 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:17:48.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:48.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:48.824 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:48.921+0000 7f0a6b46fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:48.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:48.922+0000 7f0a6b46fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:48.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:48.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:48.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.816100_664.1", 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.816100Z", 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:48.940 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:48.941 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:48.942 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:48.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:48.942 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:48Z" 2024-05-26T21:17:48.942 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:48.943 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.816100_664.1 2024-05-26T21:17:53.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:53.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:53.944 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:54.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:54.033+0000 7f0d43013a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:54.035+0000 7f0d43013a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.049 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:54.051 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:54.052 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:54.053 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:54.054 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:54.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:54.054 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:54Z" 2024-05-26T21:17:54.054 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:54.055 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.572561_648.1 2024-05-26T21:17:59.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:59.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:17:59.056 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:17:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:17:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:17:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:59.152+0000 7fb509253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:17:59.153+0000 7fb509253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:17:59.168 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:17:59.169 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:17:59.169 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:17:59.169 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:17:59.169 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:17:59.169 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:17:59.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:59.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:59.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:17:59.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:59.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:17:59.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:17:59.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:59.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:17:59.174 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:17:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:17:59.176 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:17:59Z" 2024-05-26T21:17:59.176 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:17:59.177 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:18:04.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:04.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:04.178 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:04.238 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:04.381+0000 7fe96f85ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:04.383+0000 7fe96f85ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:04.438 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:04.439 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:04.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:04.441 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.747363_723.1", 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.747363Z", 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:04.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.444 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:04.444 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:04.444 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:04.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:04.445 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:04Z" 2024-05-26T21:18:04.445 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:04.446 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756387.747363_723.1 2024-05-26T21:18:09.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:09.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:09.447 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:09.499 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:09.499 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:09.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:09.538+0000 7f9ccc20ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:09.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:09.539+0000 7f9ccc20ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:09.557 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:09.558 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:09.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:09.561 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:09.562 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:09.563 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:09.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:09.563 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:09Z" 2024-05-26T21:18:09.563 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:09.564 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:18:14.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:14.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:14.566 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:14.654+0000 7fee262e2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:14.655+0000 7fee262e2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:14.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:14.670 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:14.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:14.672 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:14.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:14Z" 2024-05-26T21:18:14.674 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:14.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:18:19.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:19.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:19.677 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:19.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:19.762+0000 7f083ceb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:19.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:19.763+0000 7f083ceb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:19.779 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:19.780 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:19.781 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:19.782 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:19.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:19.782 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:19Z" 2024-05-26T21:18:19.782 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:19.783 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:18:24.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:24.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:24.785 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:24.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:24.880+0000 7ffb76320a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:24.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:24.881+0000 7ffb76320a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.675304_834.1", 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.675304Z", 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:24Z" 2024-05-26T21:18:24.901 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:24.902 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716757758.675304_834.1 2024-05-26T21:18:29.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:29.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:29.904 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:30.115+0000 7fd3d1906a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:30.116+0000 7fd3d1906a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.132 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:30.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:30.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:30.135 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:30.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:30.137 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:30Z" 2024-05-26T21:18:30.137 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:30.138 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.648499_654.1 2024-05-26T21:18:35.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:35.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:35.139 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:35.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:35.229+0000 7f34204e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:35.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:35.230+0000 7f34204e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.243 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:35.245 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:35.246 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:35.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:35Z" 2024-05-26T21:18:35.248 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:35.249 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.633621_651.1 2024-05-26T21:18:40.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:40.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:40.250 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:40.310 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:40.311 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:40.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:40.351+0000 7f67131aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:40.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:40.352+0000 7f67131aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:40.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:40.369 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:40.370 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:40.371 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:40.372 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:40.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:40.372 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:40Z" 2024-05-26T21:18:40.372 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:40.373 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.635928_652.1 2024-05-26T21:18:45.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:45.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:45.375 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:45.433 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:45.434 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:45.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:45.473+0000 7f0acfa77a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:45.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:45.474+0000 7f0acfa77a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:45Z" 2024-05-26T21:18:45.497 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:45.498 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.677571_656.1 2024-05-26T21:18:50.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:50.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:50.500 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:50.595+0000 7fccb3a6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:50.596+0000 7fccb3a6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:50Z" 2024-05-26T21:18:50.673 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:50.673 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.632586_650.1 2024-05-26T21:18:55.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:55.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:18:55.675 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:18:55.726 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:18:55.726 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:18:55.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:55.758+0000 7f438b9caa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:55.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:18:55.759+0000 7f438b9caa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:18:55.772 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:18:55.772 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:18:55.772 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:18:55.774 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:55.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:18:55.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:18:55Z" 2024-05-26T21:18:55.777 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:18:55.778 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:19:00.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:00.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:00.780 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:00.834 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:00.834 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:00.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:00.878+0000 7f4c6a6f4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:00.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:00.879+0000 7f4c6a6f4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:00.892 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:00.892 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:00.892 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:00.892 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:00.893 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:00.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:00.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:00.896 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:00Z" 2024-05-26T21:19:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:00.898 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.820209_666.1 2024-05-26T21:19:05.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:05.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:05.900 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:05.958 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:05.958 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:05.996+0000 7fc203d84a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:05.997+0000 7fc203d84a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:06.012 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:06.013 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:06.014 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:06.015 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:05Z" 2024-05-26T21:19:06.017 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:06.017 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.797274_661.1 2024-05-26T21:19:11.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:11.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:11.019 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:11.078 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:11.078 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:11.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:11.115+0000 7f179b211a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:11.116+0000 7f179b211a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:11.130 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:19:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:11Z" 2024-05-26T21:19:11.135 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:11.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:19:16.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:16.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:16.138 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:16.189 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:16.189 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:16.222+0000 7fef68b59a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:16.223+0000 7fef68b59a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:16.237 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:16.238 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:16.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:16.242 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.725564_719.1", 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.725564Z", 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:16.243 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:16Z" 2024-05-26T21:19:16.244 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:16.246 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756387.725564_719.1 2024-05-26T21:19:21.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:21.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:21.247 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:21.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:21.332+0000 7f9207261a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:21.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:21.333+0000 7f9207261a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.347 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:21.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:21.350 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:21.351 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:21.351 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:21.351 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:21.351 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:21.354 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:21.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:21.355 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:21Z" 2024-05-26T21:19:21.355 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:21.356 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.797274_661.1 2024-05-26T21:19:26.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:26.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:26.358 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:26.417 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:26.417 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:26.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:26.453+0000 7f131638da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:26.454+0000 7f131638da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:26.471 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:26.472 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:26.473 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:26.474 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:26.475 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:26.476 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:26Z" 2024-05-26T21:19:26.477 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:26.477 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:19:31.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:31.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:31.479 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:31.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:31.574+0000 7fbc9cd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:31.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:31.575+0000 7fbc9cd6ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:31Z" 2024-05-26T21:19:31.596 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:31.597 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.820209_666.1 2024-05-26T21:19:36.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:36.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:36.598 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:36.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:36.718+0000 7f94ce271a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:36.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:36.719+0000 7f94ce271a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:36.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:36.734 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:36.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:36.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:36.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:36.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:36Z" 2024-05-26T21:19:36.739 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:36.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:19:41.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:41.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:41.743 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:41.828+0000 7f21074bda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:41.829+0000 7f21074bda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:41.843 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:19:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:41Z" 2024-05-26T21:19:41.854 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:41.856 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.652631_655.1 2024-05-26T21:19:46.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:46.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:46.857 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:46.912 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:46.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:46.947+0000 7f5acc6a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:46.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:46.948+0000 7f5acc6a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:46.961 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:46.962 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:46.963 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:46.964 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:46.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:46Z" 2024-05-26T21:19:46.966 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:46.969 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:19:51.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:51.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:51.970 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:52.065+0000 7fa851221a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:52.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:52.067+0000 7fa851221a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:52.081 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:52.082 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:52.082 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:52.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.082 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:52.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:52Z" 2024-05-26T21:19:52.087 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:52.088 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.652631_655.1 2024-05-26T21:19:57.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:57.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:19:57.089 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:19:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:19:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:19:57.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:57.176+0000 7f0f4d301a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:57.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:19:57.177+0000 7f0f4d301a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:19:57.190 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:19:57.190 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:19:57.190 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:19:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:19:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:19:57Z" 2024-05-26T21:19:57.195 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:19:57.196 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.632586_650.1 2024-05-26T21:20:02.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:02.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:02.198 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:02.252 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:02.253 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:02.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:02.285+0000 7f6c0f1a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:02.286+0000 7f6c0f1a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:02.299 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:02Z" 2024-05-26T21:20:02.305 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:02.306 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.677571_656.1 2024-05-26T21:20:07.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:07.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:07.308 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:07.367 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:07.367 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:07.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:07.408+0000 7f84e6ad2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:07.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:07.409+0000 7f84e6ad2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:07.423 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:07.424 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:07.425 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:07.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:07.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:07.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:07Z" 2024-05-26T21:20:07.429 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:07.430 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:20:12.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:12.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:12.431 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:12.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:12.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:12.528+0000 7f0ef7eeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:12.529+0000 7f0ef7eeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:12.543 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:12.545 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:12.546 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:12.547 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:12Z" 2024-05-26T21:20:12.548 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:12.549 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.731394_657.1 2024-05-26T21:20:17.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:17.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:17.551 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:17.722+0000 7fd79aa79a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:17.723+0000 7fd79aa79a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:17.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:17.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:17.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:17.741 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:17.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:17.742 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:17Z" 2024-05-26T21:20:17.742 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:17.743 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:20:22.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:22.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:22.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:22.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:22.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:22.830+0000 7f19d99eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:22.831+0000 7f19d99eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:22.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:22.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:22.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:22.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:22.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:22.849 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:22Z" 2024-05-26T21:20:22.849 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:22.850 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756387.720498_718.1 2024-05-26T21:20:27.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:27.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:27.851 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:27.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:27.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:27.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:27.936+0000 7f36c8d0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:27.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:27.937+0000 7f36c8d0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:27.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:27.953 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:27.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:27.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.956 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:27.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:27.957 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:27.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:27.957 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:27Z" 2024-05-26T21:20:27.957 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:27.959 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.652631_655.1 2024-05-26T21:20:32.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:32.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:32.960 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:33.056+0000 7fa05e558a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:33.057+0000 7fa05e558a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:33Z" 2024-05-26T21:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:33.077 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.677571_656.1 2024-05-26T21:20:38.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:38.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:38.078 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:38.174+0000 7f4011b9ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:38.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:38.175+0000 7f4011b9ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:38.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:38.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:38.195 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:38.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:38.196 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:38Z" 2024-05-26T21:20:38.196 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:38.196 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:20:43.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:43.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:43.197 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:43.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:43.381+0000 7f006a893a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:43.382+0000 7f006a893a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:43Z" 2024-05-26T21:20:43.401 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:43.402 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.635928_652.1 2024-05-26T21:20:48.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:48.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:48.404 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:48.455 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:48.455 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:48.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:48.487+0000 7f1aec7cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:48.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:48.489+0000 7f1aec7cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:48.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:48.508 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:48Z" 2024-05-26T21:20:48.508 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:48.509 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756068.652631_655.1 2024-05-26T21:20:53.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:53.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:53.510 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:53.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:53.595+0000 7fa155ef6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:53.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:53.596+0000 7fa155ef6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:53Z" 2024-05-26T21:20:53.615 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:53.616 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:20:58.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:58.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:20:58.617 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:20:58.676 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:20:58.676 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:58.713+0000 7f587b79ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:20:58.714+0000 7f587b79ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:20:58.729 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:20:58.730 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:58.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:20:58.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:20:58Z" 2024-05-26T21:20:58.734 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:20:58.735 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:21:03.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:03.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:03.736 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:03.796 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:03.797 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:03.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:03.832+0000 7f4170c8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:03.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:03.833+0000 7f4170c8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:03.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:03.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:03.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.852 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:03.853 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:03.853 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:03.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:03.853 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:03Z" 2024-05-26T21:21:03.853 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:03.854 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.572561_648.1 2024-05-26T21:21:08.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:08.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:08.855 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:08.981 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:08.981 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:09.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:09.064+0000 7f20c38a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:09.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:09.065+0000 7f20c38a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:09.084 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:09Z" 2024-05-26T21:21:09.085 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:09.086 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:21:14.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:14.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:14.088 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:14.141 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:14.141 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:14.175+0000 7f1c60778a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:14.176+0000 7f1c60778a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:14.189 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:14.189 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:14.189 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:14.190 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:14.191 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:14.192 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:14.193 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:14Z" 2024-05-26T21:21:14.195 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:14.196 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:21:19.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:19.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:19.197 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:19.255 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:19.255 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:19.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:19.291+0000 7ff097ce5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:19.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:19.293+0000 7ff097ce5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:19.307 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:19Z" 2024-05-26T21:21:19.313 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:19.313 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:21:24.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:24.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:24.316 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:24.376 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:24.376 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:24.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:24.412+0000 7f2e939c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:24.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:24.413+0000 7f2e939c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:24.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:24.432 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:24Z" 2024-05-26T21:21:24.433 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:24.435 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:21:29.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:29.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:29.436 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:29.550+0000 7f0ba906da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:29.551+0000 7f0ba906da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:29.563 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:29.563 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:29.563 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:29.563 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:29.563 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:29.564 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:29.565 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:29.566 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:29.567 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:29Z" 2024-05-26T21:21:29.568 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:29.569 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.569434_646.1 2024-05-26T21:21:34.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:34.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:34.570 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:34.655+0000 7f347f19fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:34.656+0000 7f347f19fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:34.669 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:34.669 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:34.670 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:34.672 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:34.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:34Z" 2024-05-26T21:21:34.674 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:34.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.629964_649.1 2024-05-26T21:21:39.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:39.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:39.677 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:39.729 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:39.729 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:39.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:39.765+0000 7fc0da312a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:39.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:39.766+0000 7fc0da312a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:39.779 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:39.780 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:39.781 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:21:39.782 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:39.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:39Z" 2024-05-26T21:21:39.784 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:39.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:21:44.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:44.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:44.787 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:44.882+0000 7f210ff0ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:44.883+0000 7f210ff0ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:44.897 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:44Z" 2024-05-26T21:21:44.902 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:44.903 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:21:49.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:49.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:49.905 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:50.105+0000 7f8df9e8fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:50.109+0000 7f8df9e8fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:50.143 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:50.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:50.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:50.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:50Z" 2024-05-26T21:21:50.148 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:50.149 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.633621_651.1 2024-05-26T21:21:55.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:55.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:21:55.151 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:21:55.205 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:21:55.206 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:21:55.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:55.240+0000 7f84ae33fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:55.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:21:55.241+0000 7f84ae33fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:21:55.254 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:21:55.254 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:21:55.254 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:21:55.254 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:55.255 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:55.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:21:55.258 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:21:55Z" 2024-05-26T21:21:55.259 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:21:55.260 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.570363_647.1 2024-05-26T21:22:00.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:00.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:00.262 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:00.319 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:00.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:00.358+0000 7f8f418dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:00.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:00.359+0000 7f8f418dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:00.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:00.374 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:00.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.844053_669.1", 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.844053Z", 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:00.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:00Z" 2024-05-26T21:22:00.378 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:00.379 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.844053_669.1 2024-05-26T21:22:05.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:05.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:05.381 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:05.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:05.440 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:05.478+0000 7f1bcd999a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:05.480+0000 7f1bcd999a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:05.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:05.498 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:05.501 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:05.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:05.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:05.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:05Z" 2024-05-26T21:22:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:05.505 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756387.702842_717.1 2024-05-26T21:22:10.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:10.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:10.507 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:10.661+0000 7f8453d78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:10.663+0000 7f8453d78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:10.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:10.678 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:10.679 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:10.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:10.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:10.682 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:10Z" 2024-05-26T21:22:10.682 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:10.683 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.645969_653.1 2024-05-26T21:22:15.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:15.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:15.684 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:15.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:15.769+0000 7f8f619aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:15.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:15.770+0000 7f8f619aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:15.785 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:15.785 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:15.785 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:15.785 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:15.785 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:15.786 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:15.787 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:22:15.788 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:15.789 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:15.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:15.791 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:15.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:15.791 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:15Z" 2024-05-26T21:22:15.791 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:15.792 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.572561_648.1 2024-05-26T21:22:20.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:20.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:20.793 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:20.845 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:20.845 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:20.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:20.877+0000 7f2510a03a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:20.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:20.878+0000 7f2510a03a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:20.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:20.895 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:20Z" 2024-05-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:20.897 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756067.633621_651.1 2024-05-26T21:22:25.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:25.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:25.899 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:25.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:25.995+0000 7fb043c93a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:25.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:25.996+0000 7fb043c93a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:26.011 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:26.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:25Z" 2024-05-26T21:22:26.016 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:26.017 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.797274_661.1 2024-05-26T21:22:31.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:31.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:31.018 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:31.077 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:31.077 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:31.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:31.113+0000 7fccfec1ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:31.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:31.115+0000 7fccfec1ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:31.131 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:31.132 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:31Z" 2024-05-26T21:22:31.134 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:31.135 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756069.792493_659.1 2024-05-26T21:22:36.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:36.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:36.137 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:36.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:36.271+0000 7f93b078fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:36.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:36.272+0000 7f93b078fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:36.286 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:36.286 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:36.286 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:36.286 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:36.286 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:36.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:36.288 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:36.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.769625_725.1", 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.769625Z", 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:36.290 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:36.291 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:36.292 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:36.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:36.292 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:36Z" 2024-05-26T21:22:36.292 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:36.293 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758247.756860_875.1 target=1_1716756387.769625_725.1 2024-05-26T21:22:41.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-05-26T21:22:41.296 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-61 2024-05-26T21:22:41.315 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-62 2024-05-26T21:22:41.348 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-63 2024-05-26T21:22:41.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.755 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:22:41.847 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:22:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758561.369608_903.1", 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:22:41.369608Z" 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:22:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:41.849 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:22:41.850 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:22:41.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.850 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:41.904 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:41.904 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:41.934 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:22:41.936+0000 7f3f9a093a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:41.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:22:41.937+0000 7f3f9a093a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:22:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:22:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:22:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716758561.369608_903.1", 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:22:41.369608Z", 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:22:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:22:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:22:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:22:41Z" 2024-05-26T21:22:41.954 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:22:41.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:22:41.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:22:41.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:41.956 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:42.027 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:42.028 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:42.144+0000 7fe48e8a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:42.145+0000 7fe48e8a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:42Z" 2024-05-26T21:22:42.166 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:42.167 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756068.652631_655.1 2024-05-26T21:22:47.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:47.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:47.168 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:47.225 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:47.225 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:47.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:47.260+0000 7f9161e26a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:47.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:47.261+0000 7f9161e26a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:47.275 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:47.277 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:47.278 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:47.279 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:47.280 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:47Z" 2024-05-26T21:22:47.282 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:47.282 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.635928_652.1 2024-05-26T21:22:49.529 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:22:49.529+0000 7f252a6dc700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:22:50.605 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T21:22:50.606+0000 7f30939af700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-05-26T21:22:51.767 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:22:51.768+0000 7f25326ec700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:22:52.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:52.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:52.284 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:52.503+0000 7ff7add89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:52.503+0000 7ff7add89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.517 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:52.519 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:52.520 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:52.522 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:52.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:52.522 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:52Z" 2024-05-26T21:22:52.522 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:52.523 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.800302_662.1 2024-05-26T21:22:57.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:57.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:22:57.524 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:22:57.574 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:22:57.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:22:57.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:57.607+0000 7ff515ec8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:57.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:22:57.608+0000 7ff515ec8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:22:57.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:57.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:22:57.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:22:57Z" 2024-05-26T21:22:57.626 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:22:57.627 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:23:02.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:02.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:02.628 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:02.680 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:02.681 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:02.714+0000 7ff2c0860a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:02.715+0000 7ff2c0860a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:02.728 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:02Z" 2024-05-26T21:23:02.733 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:02.734 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:23:07.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:07.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:07.735 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:07.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:07.828+0000 7f52e5a90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:07.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:07.829+0000 7f52e5a90a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:07Z" 2024-05-26T21:23:07.849 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:07.850 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756068.652631_655.1 2024-05-26T21:23:12.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:12.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:12.852 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:12.911 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:12.911 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:12.947+0000 7fe39f3eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:12.948+0000 7fe39f3eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:12Z" 2024-05-26T21:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:12.968 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:23:17.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:17.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:17.970 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:18.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:18.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:18.146+0000 7fe9301eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:18.147+0000 7fe9301eea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:18.160 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:18.160 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:18.160 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:18.160 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:18.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:18.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:18Z" 2024-05-26T21:23:18.165 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:18.166 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.645969_653.1 2024-05-26T21:23:23.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:23.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:23.168 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:23.222 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:23.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:23.256+0000 7f239d3d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:23.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:23.257+0000 7f239d3d2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:23.270 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:23.270 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:23.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:23.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:23.274 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:23Z" 2024-05-26T21:23:23.275 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:23.276 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.632586_650.1 2024-05-26T21:23:28.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:28.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:28.278 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:28.366+0000 7fa1ba0b9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:28.367+0000 7fa1ba0b9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:28.380 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:28.381 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:28.382 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:23:28.383 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:28.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:28Z" 2024-05-26T21:23:28.385 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:28.386 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.570363_647.1 2024-05-26T21:23:33.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:33.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:33.388 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:33.486+0000 7f32ddb0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:33.487+0000 7f32ddb0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:33.501 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:33.502 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:33.502 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:33.502 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:33.502 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:33.502 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:33.503 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:33.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:33.505 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:33.506 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:33.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:33.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:33.509 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:33.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:33.511 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:33Z" 2024-05-26T21:23:33.512 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:33.513 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.731394_657.1 2024-05-26T21:23:38.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:38.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:38.515 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:38.610+0000 7fac7d7b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:38.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:38.611+0000 7fac7d7b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:38.626 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:38Z" 2024-05-26T21:23:38.631 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:38.632 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.795957_660.1 2024-05-26T21:23:43.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:43.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:43.634 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:43.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:43.841+0000 7f3c878e0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:43.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:43.855+0000 7f3c878e0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.816100_664.1", 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.816100Z", 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:43Z" 2024-05-26T21:23:43.894 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:43.895 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.816100_664.1 2024-05-26T21:23:48.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:48.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:48.896 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:48.952 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:48.952 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:48.983+0000 7f81d23bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:48.984+0000 7f81d23bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:48.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:49.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:49.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:49.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:48Z" 2024-05-26T21:23:49.003 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:49.004 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.735465_658.1 2024-05-26T21:23:54.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:54.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:54.005 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:54.099+0000 7fafb9413a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:54.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:54.100+0000 7fafb9413a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:54.115 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:54Z" 2024-05-26T21:23:54.120 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:54.121 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.632586_650.1 2024-05-26T21:23:59.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:59.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:23:59.122 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:23:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:23:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:23:59.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:59.222+0000 7f34c92b1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:59.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:23:59.223+0000 7f34c92b1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:23:59.237 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:23:59.239 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:59.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:23:59.241 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:23:59.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:23:59.243 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:23:59Z" 2024-05-26T21:23:59.243 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:23:59.243 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.735465_658.1 2024-05-26T21:24:04.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:04.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:04.245 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:04.305 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:04.306 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:04.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:04.346+0000 7ff1adc32a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:04.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:04.353+0000 7ff1adc32a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:04.468 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:04.468 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:04.468 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:04.468 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:04.468 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:04.469 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:04.470 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:04.471 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:04.472 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.473 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:04.474 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:04.474 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:04.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:04.474 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:04Z" 2024-05-26T21:24:04.474 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:04.474 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.629964_649.1 2024-05-26T21:24:09.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:09.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:09.476 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:09.560+0000 7f114c46fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:09.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:09.561+0000 7f114c46fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:09.574 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:09.574 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:09.575 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:09.576 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:09.577 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.725564_719.1", 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.725564Z", 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:09.578 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:09.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:09Z" 2024-05-26T21:24:09.580 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:09.581 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.725564_719.1 2024-05-26T21:24:14.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:14.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:14.583 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:14.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:14.667+0000 7f154745da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:14.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:14.668+0000 7f154745da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:14.682 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:14.682 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:14.682 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:14.682 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:14.682 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:14.683 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:14.684 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:14.685 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:14.686 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:14Z" 2024-05-26T21:24:14.687 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:14.688 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.569434_646.1 2024-05-26T21:24:19.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:19.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:19.690 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:19.777+0000 7ff826382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:19.778+0000 7ff826382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:19.792 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:19.793 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:19.794 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:19.795 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:19.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:19.797 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:19Z" 2024-05-26T21:24:19.797 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:19.798 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.570363_647.1 2024-05-26T21:24:24.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:24.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:24.799 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:24.858 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:24.858 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:24.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:24.894+0000 7f7c78baba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:24.895+0000 7f7c78baba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.910 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:24.912 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:24.913 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:24.914 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:24.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:24.915 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:24Z" 2024-05-26T21:24:24.915 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:24.916 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.629964_649.1 2024-05-26T21:24:29.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:29.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:29.918 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:29.971 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:29.972 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:30.005+0000 7f3be10c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:30.005+0000 7f3be10c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.019 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:30.021 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:24:30.022 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:30.023 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:30Z" 2024-05-26T21:24:30.024 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:30.025 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.570363_647.1 2024-05-26T21:24:35.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:35.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:35.027 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:35.111+0000 7fc5c106ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:35.112+0000 7fc5c106ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:35.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:35.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:35.128 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:35.128 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:35.132 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:35.133 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:35.134 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:35Z" 2024-05-26T21:24:35.135 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:35.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:24:40.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:40.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:40.138 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:40.233+0000 7f0c5f932a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:40.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:40.234+0000 7f0c5f932a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:40.248 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:40.250 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:40.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:40Z" 2024-05-26T21:24:40.253 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:40.254 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.702842_717.1 2024-05-26T21:24:45.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:45.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:45.256 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:45.315 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:45.316 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:45.352+0000 7fd4d8faaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:45.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:45.353+0000 7fd4d8faaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:45Z" 2024-05-26T21:24:45.373 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:45.373 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.569434_646.1 2024-05-26T21:24:50.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:50.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:50.375 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:50.452 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:50.453 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:50.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:50.592+0000 7f5ac6d05a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:50.594+0000 7f5ac6d05a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:50.607 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:50.607 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:50.607 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:50.607 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:50.607 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:50.608 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:50.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:24:50.610 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:50.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:50Z" 2024-05-26T21:24:50.612 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:50.613 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.645969_653.1 2024-05-26T21:24:55.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:55.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:24:55.615 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:24:55.670 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:24:55.670 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:24:55.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:55.704+0000 7fd927d0aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:55.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:24:55.705+0000 7fd927d0aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:24:55.719 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:24:55.719 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:24:55.719 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:24:55.719 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:24:55.719 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:24:55.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:55.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.723 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:24:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:24:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:24:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:24:55.724 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:24:55Z" 2024-05-26T21:24:55.724 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:24:55.725 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.569434_646.1 2024-05-26T21:25:00.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:00.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:00.726 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:00.782 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:00.782 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:00.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:00.816+0000 7f4f4ffdaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:00.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:00.817+0000 7f4f4ffdaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:00.832 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:00.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:00.834 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.730169_720.1", 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.730169Z", 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:00.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:00.836 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:00Z" 2024-05-26T21:25:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:00.838 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.730169_720.1 2024-05-26T21:25:05.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:05.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:05.840 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:05.899 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:05.899 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:05.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:05.935+0000 7fe56f205a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:05.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:05.936+0000 7fe56f205a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:05.951 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:05.952 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:05.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:05.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:05.955 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:05Z" 2024-05-26T21:25:05.956 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:05.957 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756068.652631_655.1 2024-05-26T21:25:10.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:10.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:10.959 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:11.035 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:11.035 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:11.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:11.071+0000 7ff94a1cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:11.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:11.072+0000 7ff94a1cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:11.085 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.747363_723.1", 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.747363Z", 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:11.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:11.094 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:11.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:11.094 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:11Z" 2024-05-26T21:25:11.094 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:11.095 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.747363_723.1 2024-05-26T21:25:16.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:16.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:16.096 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:16.147 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:16.147 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:16.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:16.179+0000 7ff3330a0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:16.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:16.180+0000 7ff3330a0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:16.193 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:16.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:16.195 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:16.196 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:16.197 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:16Z" 2024-05-26T21:25:16.198 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:16.199 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.720498_718.1 2024-05-26T21:25:21.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:21.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:21.201 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:21.259 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:21.296+0000 7fa96f4a1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:21.297+0000 7fa96f4a1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:21.311 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:21.312 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:21.313 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:25:21.314 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:21.315 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:21Z" 2024-05-26T21:25:21.316 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:21.317 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756068.652631_655.1 2024-05-26T21:25:26.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:26.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:26.319 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:26.379 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:26.379 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:26.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:26.416+0000 7f9d8a825a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:26.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:26.417+0000 7f9d8a825a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:26.431 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:26.432 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:26.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:25:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:26.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:26.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:26.437 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:26Z" 2024-05-26T21:25:26.437 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:26.437 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.645969_653.1 2024-05-26T21:25:31.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:31.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:31.439 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:31.498 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:31.498 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:31.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:31.537+0000 7fcb18620a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:31.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:31.545+0000 7fcb18620a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:31.620 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:31Z" 2024-05-26T21:25:31.625 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:31.626 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.635928_652.1 2024-05-26T21:25:36.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:36.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:36.627 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:36.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:36.712+0000 7fe8a29caa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:36.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:36.713+0000 7fe8a29caa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:36.728 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:36.729 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:36.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:36Z" 2024-05-26T21:25:36.732 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:36.733 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:25:41.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:41.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:41.734 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:41.827+0000 7f109a355a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:41.828+0000 7f109a355a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:41.842 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:41Z" 2024-05-26T21:25:41.851 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:41.852 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:25:46.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:46.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:46.854 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:46.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:46.949+0000 7fba16271a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:46.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:46.951+0000 7fba16271a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:46.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:46.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:46.969 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:46.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:46Z" 2024-05-26T21:25:46.971 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:46.972 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756068.652631_655.1 2024-05-26T21:25:51.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:51.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:51.973 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:52.041 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:52.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:52.176+0000 7f156cb20a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:52.180+0000 7f156cb20a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:52.218 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:52.219 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:52.220 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.773172_727.1", 2024-05-26T21:25:52.221 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.773172Z", 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:52.222 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:52.223 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:52.224 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:52Z" 2024-05-26T21:25:52.224 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:52.225 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.773172_727.1 2024-05-26T21:25:57.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:57.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:25:57.226 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:25:57.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:25:57.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:25:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:57.312+0000 7f4982ef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:25:57.312+0000 7f4982ef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:25:57.325 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:25:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:25:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:25:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:25:57.325 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:57.326 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:57.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:25:57.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:57.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:25:57Z" 2024-05-26T21:25:57.330 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:25:57.331 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.572561_648.1 2024-05-26T21:26:02.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:02.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:02.333 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:02.385 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:02.385 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:02.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:02.417+0000 7f5ff164aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:02.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:02.418+0000 7f5ff164aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:02.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:02.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:02.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:02.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:02Z" 2024-05-26T21:26:02.437 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:02.438 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.633621_651.1 2024-05-26T21:26:07.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:07.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:07.439 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:07.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:07.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:07.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:07.521+0000 7f3781336a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:07.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:07.523+0000 7f3781336a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:07.536 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:07.536 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:07.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:07.538 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:07.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:07Z" 2024-05-26T21:26:07.541 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:07.542 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.792493_659.1 2024-05-26T21:26:12.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:12.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:12.544 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:12.596 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:12.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:12.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:12.628+0000 7fc15bb16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:12.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:12.629+0000 7fc15bb16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:12.642 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:12.642 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:12.642 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:12.642 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:12.643 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:26:12.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:12.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:12Z" 2024-05-26T21:26:12.647 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:12.648 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.797274_661.1 2024-05-26T21:26:17.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:17.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:17.649 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:17.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:17.746+0000 7f29cfbeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:17.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:17.747+0000 7f29cfbeda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:17.761 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:17.761 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:17.761 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:17.761 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:17.761 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:17.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:17.765 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:17Z" 2024-05-26T21:26:17.768 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:17.769 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.648499_654.1 2024-05-26T21:26:22.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:22.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:22.770 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:22.829 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:22.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:22.865+0000 7f907a4e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:22.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:22.867+0000 7f907a4e8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:22.881 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:22.881 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:22.882 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:22.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:22.884 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:22.885 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:22Z" 2024-05-26T21:26:22.886 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:22.887 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.632586_650.1 2024-05-26T21:26:27.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:27.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:27.888 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:27.947 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:27.947 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:27.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:27.982+0000 7f2aa4c83a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:27.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:27.983+0000 7f2aa4c83a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:27.998 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:27.998 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:27.998 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:27.999 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:27Z" 2024-05-26T21:26:28.003 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:28.004 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.795957_660.1 2024-05-26T21:26:33.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:33.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:33.005 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:33.096 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:33.097 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:33.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:33.147+0000 7fe8bbaa4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:33.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:33.148+0000 7fe8bbaa4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:33Z" 2024-05-26T21:26:33.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:33.168 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.632586_650.1 2024-05-26T21:26:38.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:38.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:38.169 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:38.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:38.254+0000 7f4ab8e5fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:38.255+0000 7f4ab8e5fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:38.268 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:38.268 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:38.268 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:38.268 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:38.268 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:38.269 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.272 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:38Z" 2024-05-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:38.274 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.570363_647.1 2024-05-26T21:26:43.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:43.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:43.275 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:43.365+0000 7f6766171a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:43.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:43.365+0000 7f6766171a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:43.379 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:26:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:43Z" 2024-05-26T21:26:43.387 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:43.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.720498_718.1 2024-05-26T21:26:48.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:48.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:48.390 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:48.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:48.486+0000 7f8adc6c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:48.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:48.487+0000 7f8adc6c9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:48.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:48.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:48.504 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:48.505 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.506 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:48Z" 2024-05-26T21:26:48.507 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:48.508 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.702842_717.1 2024-05-26T21:26:53.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:53.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:53.509 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:53.715+0000 7f0a129aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:53.718+0000 7f0a129aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:53.731 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:26:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:53Z" 2024-05-26T21:26:53.736 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:53.737 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.632586_650.1 2024-05-26T21:26:58.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:58.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:26:58.739 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:26:58.791 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:26:58.791 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:26:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:58.823+0000 7fe16f882a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:58.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:26:58.824+0000 7fe16f882a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:26:58.837 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:26:58.837 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:26:58.837 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:26:58.838 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:26:58.839 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:26:58.840 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:26:58.841 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.842 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:26:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:26:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:26:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:26:58.843 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:26:58Z" 2024-05-26T21:26:58.843 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:26:58.844 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756387.702842_717.1 2024-05-26T21:27:03.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:03.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:03.845 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:03.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:03.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:03.942+0000 7f34aa4dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:03.942+0000 7f34aa4dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:03.956 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:03.956 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:03.956 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:03.956 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:03.956 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:03.957 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:27:03.959 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:03.960 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:03Z" 2024-05-26T21:27:03.961 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:03.962 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.569434_646.1 2024-05-26T21:27:08.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:08.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:08.964 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:09.023 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:09.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:09.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:09.061+0000 7f6c483f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:09.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:09.062+0000 7f6c483f6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.817362_665.1", 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.817362Z", 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.081 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:09Z" 2024-05-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:09.083 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.817362_665.1 2024-05-26T21:27:14.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:14.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:14.085 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:14.143 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:14.143 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:14.246+0000 7fe6fa660a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:14.250+0000 7fe6fa660a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:14.316 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:14Z" 2024-05-26T21:27:14.321 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:14.322 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.569434_646.1 2024-05-26T21:27:19.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:19.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:19.323 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:19.406+0000 7f7fd7c64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:19.407+0000 7f7fd7c64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:19.424 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:19.424 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:19.424 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:19.424 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:19.425 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:19.426 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:27:19.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:19.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:19Z" 2024-05-26T21:27:19.429 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:19.430 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.635928_652.1 2024-05-26T21:27:24.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:24.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:24.432 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:24.482 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:24.482 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:24.517+0000 7f26cb606a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:24.518+0000 7f26cb606a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:24.531 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:24.532 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:24.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:24.534 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:24.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:24Z" 2024-05-26T21:27:24.536 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:24.537 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756069.813012_663.1 2024-05-26T21:27:29.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:29.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:29.538 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:29.592 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:29.592 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:29.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:29.626+0000 7f9eeb3c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:29.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:29.627+0000 7f9eeb3c8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:29.640 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:29.640 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:29.641 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:29.642 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:27:29.643 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:29.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.645 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:29.646 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:29.646 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:29.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:29.646 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:29Z" 2024-05-26T21:27:29.646 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:29.647 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.648499_654.1 2024-05-26T21:27:34.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:34.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:34.649 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:34.708 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:34.744+0000 7f2b2d297a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:34.745+0000 7f2b2d297a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:34.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:34.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:34.763 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:34.764 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:34.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:34.765 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:34Z" 2024-05-26T21:27:34.765 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:34.766 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.633621_651.1 2024-05-26T21:27:39.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:39.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:39.767 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:39.865+0000 7f622853da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:39.866+0000 7f622853da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.881 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:39.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:39.883 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:39.884 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:39.885 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:39.886 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:39.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:39.886 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:39Z" 2024-05-26T21:27:39.886 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:39.887 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.645969_653.1 2024-05-26T21:27:44.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:44.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:44.888 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:44.989 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:44.989 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:45.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:45.056+0000 7f017f330a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:45.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:45.057+0000 7f017f330a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:45Z" 2024-05-26T21:27:45.076 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:45.077 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758561.369608_903.1 target=1_1716756067.570363_647.1 2024-05-26T21:27:50.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-05-26T21:27:50.079 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-64 2024-05-26T21:27:50.100 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-65 2024-05-26T21:27:50.136 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xmqxus-66 2024-05-26T21:27:51.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.773 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:51.826 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:51.826 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:51.857 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:27:51.874 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758870.157992_924.1", 2024-05-26T21:27:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:27:50.157992Z" 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:51.876 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:27:51.876 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:27:51.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.877 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:51.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:51.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:51.965 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:27:51.966+0000 7f37cc40da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:51.966 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:27:51.966+0000 7f37cc40da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:27:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:27:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716758870.157992_924.1", 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:27:50.157992Z", 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:27:51.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:27:51.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:27:51Z" 2024-05-26T21:27:51.984 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:27:51.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:27:51.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:27:51.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:51.985 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:52.071+0000 7fa66af98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:52.072+0000 7fa66af98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:52.085 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:52.085 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:52.086 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:52.087 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:52.088 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:52.089 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:52Z" 2024-05-26T21:27:52.090 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:52.091 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:27:57.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:57.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:27:57.093 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:27:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:27:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:27:57.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:57.177+0000 7fa093668a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:57.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:27:57.178+0000 7fa093668a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:27:57.191 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:27:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:27:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:57.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:57.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:27:57.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:27:57.196 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:27:57.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:27:57.196 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:27:57Z" 2024-05-26T21:27:57.196 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:27:57.197 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.633621_651.1 2024-05-26T21:28:02.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:02.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:02.198 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:02.250 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:02.250 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:02.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:02.285+0000 7f4156db9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:02.286+0000 7f4156db9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:02.299 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:02.300 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:02.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.303 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:02.304 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:02Z" 2024-05-26T21:28:02.304 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:02.305 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:28:07.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:07.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:07.306 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:07.383 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:07.383 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:07.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:07.420+0000 7f9e28127a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:07.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:07.421+0000 7f9e28127a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:07.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.439 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:07.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:07.440 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:07.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:07.440 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:07Z" 2024-05-26T21:28:07.440 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:07.441 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.652631_655.1 2024-05-26T21:28:09.580 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:28:09.581+0000 7f2530ee9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:28:12.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:12.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:12.443 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:12.493 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:12.493 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:12.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:12.525+0000 7f5b78382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:12.526+0000 7f5b78382a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:12.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:12.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:12.541 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.817362_665.1", 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.817362Z", 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:12.542 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:12.544 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:12Z" 2024-05-26T21:28:12.544 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:12.545 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.817362_665.1 2024-05-26T21:28:17.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:17.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:17.546 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:17.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:17.675+0000 7f85e358ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:17.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:17.689+0000 7f85e358ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:17.765 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:17.766 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:17.767 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:17.768 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:17.769 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:17.771 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:17.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:17.771 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:17Z" 2024-05-26T21:28:17.771 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:17.772 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756387.720498_718.1 2024-05-26T21:28:22.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:22.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:22.773 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:22.858+0000 7fc9a7a3aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:22.859+0000 7fc9a7a3aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:22.872 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:22.872 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:22.872 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:22.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:22.874 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:28:22.875 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:22.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:22Z" 2024-05-26T21:28:22.877 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:22.878 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.572561_648.1 2024-05-26T21:28:27.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:27.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:27.879 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:27.937 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:27.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:27.972+0000 7f60b6409a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:27.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:27.973+0000 7f60b6409a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:27.987 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:27.988 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:27.989 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:27.990 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:27.991 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:27.992 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:27.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:27.993 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:27Z" 2024-05-26T21:28:27.993 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:27.994 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.629964_649.1 2024-05-26T21:28:32.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:32.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:32.995 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:33.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:33.079+0000 7f632f4e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:33.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:33.080+0000 7f632f4e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:33Z" 2024-05-26T21:28:33.099 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:33.100 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:28:38.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:38.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:38.101 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:38.193+0000 7ffabce7ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:38.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:38.194+0000 7ffabce7ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:38.208 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:38.208 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:38.208 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:38.208 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:38.208 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:38.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:38.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:38.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:38Z" 2024-05-26T21:28:38.213 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:38.214 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.677571_656.1 2024-05-26T21:28:43.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:43.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:43.216 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:43.266 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:43.266 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:43.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:43.300+0000 7f851952ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:43.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:43.301+0000 7f851952ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:43Z" 2024-05-26T21:28:43.320 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:43.321 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:28:48.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:48.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:48.323 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:48.411+0000 7faa55ac9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:48.412+0000 7faa55ac9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:48.426 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:48.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:48.430 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:48.431 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:48.432 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:48.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:48.432 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:48Z" 2024-05-26T21:28:48.432 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:48.433 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.572561_648.1 2024-05-26T21:28:53.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:53.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:53.434 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:53.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:53.523+0000 7f2bbf1ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:53.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:53.524+0000 7f2bbf1ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:53.538 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:53.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:53.540 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:53.541 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:53.543 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:53.543 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:53.546 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:53.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:53.546 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:53Z" 2024-05-26T21:28:53.546 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:53.547 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.632586_650.1 2024-05-26T21:28:58.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:58.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:28:58.548 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:28:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:28:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:28:58.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:58.742+0000 7f182556ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:28:58.746+0000 7f182556ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:28:58.773 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:28:58.773 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:58.774 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:28:58.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:28:58.776 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:28:58.777 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:28:58Z" 2024-05-26T21:28:58.778 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:28:58.779 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.645969_653.1 2024-05-26T21:29:03.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:03.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:03.781 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:03.833 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:03.833 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:03.865+0000 7f0a5ac94a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:03.866+0000 7f0a5ac94a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:03.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:03.881 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:03.882 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:03.883 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:03.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:03.885 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:03Z" 2024-05-26T21:29:03.885 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:03.885 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.813012_663.1 2024-05-26T21:29:08.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:08.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:08.888 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:08.947 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:08.947 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:08.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:08.983+0000 7ff44e20aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:08.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:08.984+0000 7ff44e20aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:08.999 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:08.999 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:08.999 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:08.999 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:08.999 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:09.000 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:09.001 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:09.002 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:09.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:08Z" 2024-05-26T21:29:09.004 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:09.005 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:29:14.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:14.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:14.006 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:14.116 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:14.117 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:14.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:14.194+0000 7f1d4f3f8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:14.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:14.195+0000 7f1d4f3f8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:14.209 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:14.210 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:14.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:14.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:14.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:14Z" 2024-05-26T21:29:14.214 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:14.215 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.652631_655.1 2024-05-26T21:29:19.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:19.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:19.216 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:19.274 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:19.274 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:19.324+0000 7fc5afe4ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:19.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:19.327+0000 7fc5afe4ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:19.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:19.346 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.720498_718.1", 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.720498Z", 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:19Z" 2024-05-26T21:29:19.350 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:19.351 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756387.720498_718.1 2024-05-26T21:29:24.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:24.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:24.352 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:24.405 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:24.406 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:24.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:24.441+0000 7f0a052ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:24.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:24.442+0000 7f0a052ada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:24.456 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:24.457 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:24.458 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:24.459 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:24.460 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:24.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:24.462 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:24Z" 2024-05-26T21:29:24.462 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:24.463 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.648499_654.1 2024-05-26T21:29:29.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:29.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:29.464 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:29.523 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:29.523 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:29.559+0000 7f4c5b35da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:29.560+0000 7f4c5b35da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.575 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:29.576 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:29.577 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:29.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:29Z" 2024-05-26T21:29:29.580 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:29.582 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.645969_653.1 2024-05-26T21:29:34.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:34.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:34.584 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:34.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:34.679+0000 7f93b58c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:34.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:34.680+0000 7f93b58c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:34.694 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:34.694 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:34.694 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:34.695 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:34.696 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:29:34.697 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:34.698 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:34.699 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:34Z" 2024-05-26T21:29:34.700 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:34.700 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.731394_657.1 2024-05-26T21:29:39.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:39.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:39.702 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:39.805 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:39.805 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:39.864+0000 7f811aef9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:39.865+0000 7f811aef9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:39.879 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:39.879 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:39.879 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:39.945 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:39.945 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:39.945 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:39.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:39.947 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.769625_725.1", 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.769625Z", 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:39.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:39.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:39Z" 2024-05-26T21:29:39.950 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:39.951 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756387.769625_725.1 2024-05-26T21:29:44.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:44.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:44.952 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:45.005 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:45.005 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:45.037+0000 7f0b94a16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:45.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:45.038+0000 7f0b94a16a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:45Z" 2024-05-26T21:29:45.057 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:45.058 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.731394_657.1 2024-05-26T21:29:50.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:50.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:50.060 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:50.156+0000 7f6a95de0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:50.157+0000 7f6a95de0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:50.171 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:50.171 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:50.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:50.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.731394_657.1", 2024-05-26T21:29:50.174 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.731394Z", 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:50.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:50.177 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:50Z" 2024-05-26T21:29:50.177 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:50.178 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.731394_657.1 2024-05-26T21:29:55.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:55.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:29:55.179 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:29:55.241 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:29:55.242 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:29:55.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:55.277+0000 7fd2cc21ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:55.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:29:55.278+0000 7fd2cc21ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:29:55.292 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:29:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:55.294 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:29:55.295 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:29:55.296 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:29:55.297 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:29:55.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:29:55.298 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:29:55Z" 2024-05-26T21:29:55.298 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:29:55.299 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:30:00.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:00.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:00.300 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:00.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:00.419+0000 7f9d4ebfca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:00.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:00.420+0000 7f9d4ebfca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:00.433 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:00.434 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:00.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:30:00.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:00.437 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:00Z" 2024-05-26T21:30:00.438 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:00.439 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.645969_653.1 2024-05-26T21:30:05.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:05.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:05.441 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:05.495 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:05.526+0000 7fe338751a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:05.527+0000 7fe338751a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:05.541 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:05.542 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:05.543 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:05.544 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:05.545 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:05.546 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:05.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:05.546 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:05Z" 2024-05-26T21:30:05.546 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:05.547 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.632586_650.1 2024-05-26T21:30:10.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:10.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:10.548 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:10.637+0000 7f7b086dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:10.638+0000 7f7b086dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:10.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:10.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:10.726 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:10.727 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:10.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:10Z" 2024-05-26T21:30:10.729 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:10.730 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.635928_652.1 2024-05-26T21:30:15.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:15.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:15.731 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:15.789 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:15.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:15.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:15.826+0000 7ff6baa80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:15.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:15.827+0000 7ff6baa80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:15.842 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:15.842 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:15.842 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:15.842 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:15.842 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:15.843 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:15.844 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:15Z" 2024-05-26T21:30:15.847 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:15.848 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.572561_648.1 2024-05-26T21:30:20.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:20.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:20.849 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:21.047+0000 7fce0c05ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:21.048+0000 7fce0c05ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:21.063 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:21.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:21.066 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.846685_670.1", 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:21.067 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.846685Z", 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:21.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:21Z" 2024-05-26T21:30:21.069 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:21.070 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.846685_670.1 2024-05-26T21:30:26.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:26.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:26.071 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:26.124 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:26.124 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:26.156+0000 7f4d54277a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:26.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:26.157+0000 7f4d54277a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:26.170 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:26.170 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:26.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:26.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:26.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:26.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:26Z" 2024-05-26T21:30:26.175 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:26.176 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.677571_656.1 2024-05-26T21:30:31.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:31.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:31.178 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:31.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:31.267+0000 7f7c794e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:31.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:31.268+0000 7f7c794e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:31.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:31.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:31.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:31.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:31Z" 2024-05-26T21:30:31.287 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:31.288 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:30:36.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:36.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:36.289 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:36.350 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:36.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:36.388+0000 7f4557ef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:36.389+0000 7f4557ef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:36.403 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:36.404 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:36.405 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:36.406 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:36.407 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:36Z" 2024-05-26T21:30:36.408 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:36.409 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:30:41.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:41.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:41.411 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:41.482 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:41.515+0000 7fb60625fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:41.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:41.516+0000 7fb60625fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:41.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:41.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:41.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:41.534 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.535 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:41.536 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:41.536 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:41.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:41.536 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:41Z" 2024-05-26T21:30:41.536 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:41.538 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.633621_651.1 2024-05-26T21:30:46.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:46.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:46.539 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:46.593 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:46.594 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:46.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:46.625+0000 7f6c14ad5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:46.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:46.626+0000 7f6c14ad5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:46.639 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:46.639 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:46.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:46.641 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:46.642 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:46.643 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:46Z" 2024-05-26T21:30:46.644 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:46.645 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.797274_661.1 2024-05-26T21:30:51.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:51.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:51.648 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:51.702 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:51.702 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:51.734+0000 7fd638db3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:51.735+0000 7fd638db3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:51.748 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:51.749 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:51.751 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.629964_649.1", 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:51.752 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.629964Z", 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:51.753 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:51.754 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:51.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:51.755 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:51Z" 2024-05-26T21:30:51.755 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:51.756 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.629964_649.1 2024-05-26T21:30:56.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:56.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:30:56.757 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:30:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:30:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:30:56.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:56.853+0000 7f70d2975a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:56.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:30:56.854+0000 7f70d2975a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:30:56.869 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:56.870 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:56.871 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.816100_664.1", 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:56.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.816100Z", 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:56.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:30:56Z" 2024-05-26T21:30:56.874 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:30:56.875 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.816100_664.1 2024-05-26T21:31:01.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:01.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:01.877 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:01.979 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:01.979 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:02.129+0000 7fcec292ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:02.130+0000 7fcec292ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:02.143 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:02.143 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:02.144 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:02.145 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:02.146 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:02.147 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:02.147 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:02.147 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.842846_668.1", 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.842846Z", 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:02.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:02Z" 2024-05-26T21:31:02.152 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:02.153 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.842846_668.1 2024-05-26T21:31:07.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:07.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:07.154 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:07.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:07.243+0000 7f7fb4aeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:07.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:07.244+0000 7f7fb4aeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:07.257 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.652631_655.1", 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.652631Z", 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:07Z" 2024-05-26T21:31:07.264 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:07.265 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.652631_655.1 2024-05-26T21:31:12.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:12.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:12.267 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:12.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:12.354+0000 7f3f16e91a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:12.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:12.355+0000 7f3f16e91a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:12.368 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:12.369 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:12.370 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:12.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:12.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:12.373 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:12Z" 2024-05-26T21:31:12.374 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:12.374 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.635928_652.1 2024-05-26T21:31:17.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:17.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:17.376 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:17.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:17.474+0000 7f9b45e76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:17.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:17.475+0000 7f9b45e76a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:17.491 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:17.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:17.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:17.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:17Z" 2024-05-26T21:31:17.496 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:17.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:31:22.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:22.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:22.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:22.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:22.708+0000 7fc57b47ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:22.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:22.710+0000 7fc57b47ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:22.732 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:22.732 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:22.732 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:22.732 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:22.732 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:22.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:22.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.633621_651.1", 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.633621Z", 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:22.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:22.739 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:22.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:22.740 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:22Z" 2024-05-26T21:31:22.740 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:22.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.633621_651.1 2024-05-26T21:31:27.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:27.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:27.742 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:27.830+0000 7f6875ae2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:27.831+0000 7f6875ae2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:27.844 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:27Z" 2024-05-26T21:31:27.849 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:27.850 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.635928_652.1 2024-05-26T21:31:32.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:32.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:32.852 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:32.938+0000 7f9572424a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:32.939+0000 7f9572424a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:32.952 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:32.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:32.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:32.955 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:32.956 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:32Z" 2024-05-26T21:31:32.957 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:32.958 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:31:37.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:37.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:37.960 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:38.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:38.057+0000 7f99e375ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:38.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:38.058+0000 7f99e375ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.570363_647.1", 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.570363Z", 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:38Z" 2024-05-26T21:31:38.078 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:38.079 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.570363_647.1 2024-05-26T21:31:43.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:43.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:43.081 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:43.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:43.176+0000 7f30edeada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:43.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:43.177+0000 7f30edeada80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:43.191 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.645969_653.1", 2024-05-26T21:31:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.645969Z", 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:43Z" 2024-05-26T21:31:43.196 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:43.197 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.645969_653.1 2024-05-26T21:31:48.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:48.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:48.199 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:48.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:48.360+0000 7f2910deda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:48.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:48.361+0000 7f2910deda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:48.375 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:48.375 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:48.375 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:48.375 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:48.375 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:48.376 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:48.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756705.929260_766.1", 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:48.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:51:45.929260Z", 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:48.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:48.380 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:48.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:48.381 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:48Z" 2024-05-26T21:31:48.381 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:48.382 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756705.929260_766.1 2024-05-26T21:31:53.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:53.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:53.384 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:53.468+0000 7f1f4d541a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:53.469+0000 7f1f4d541a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:53.482 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:53.482 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:53.482 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:53.482 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:53.482 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:53.484 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:53Z" 2024-05-26T21:31:53.487 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:53.488 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.572561_648.1 2024-05-26T21:31:58.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:58.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:31:58.491 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:31:58.544 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:31:58.544 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:31:58.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:58.576+0000 7fd329ca7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:31:58.577+0000 7fd329ca7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:58.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:31:58.593 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.594 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:31:58.595 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:31:58.595 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:31:58.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:31:58.595 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:31:58Z" 2024-05-26T21:31:58.595 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:31:58.596 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:32:03.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:03.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:03.598 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:03.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:03.691+0000 7f8f6b8e9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:03.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:03.692+0000 7f8f6b8e9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:03Z" 2024-05-26T21:32:03.713 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:03.713 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.632586_650.1 2024-05-26T21:32:08.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:08.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:08.715 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:08.811+0000 7f824b8faa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:08.812+0000 7f824b8faa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:08.828 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:08.829 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:08.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:08.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:08.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:08Z" 2024-05-26T21:32:08.833 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:08.834 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:32:13.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:13.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:13.836 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:13.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:13.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:13.980+0000 7f51d0bc6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:13.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:13.981+0000 7f51d0bc6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:13.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:13.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:13.998 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:13.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:13Z" 2024-05-26T21:32:14.000 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:14.001 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.648499_654.1 2024-05-26T21:32:19.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:19.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:19.002 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:19.056 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:19.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:19.088+0000 7f0b6cd78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:19.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:19.089+0000 7f0b6cd78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:19.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:19.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:19.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:19.107 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:19Z" 2024-05-26T21:32:19.108 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:19.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.632586_650.1 2024-05-26T21:32:24.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:24.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:24.110 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:24.169 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:24.170 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:24.206+0000 7f40f6764a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:24.207+0000 7f40f6764a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:24.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.846685_670.1", 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.846685Z", 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:24.229 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:24Z" 2024-05-26T21:32:24.230 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:24.231 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756069.846685_670.1 2024-05-26T21:32:29.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:29.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:29.234 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:29.328+0000 7fc9fc9fea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:29.329+0000 7fc9fc9fea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:29.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:29.345 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:29.346 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:29.349 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:29Z" 2024-05-26T21:32:29.349 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:29.350 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.635928_652.1 2024-05-26T21:32:34.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:34.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:34.352 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:34.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:34.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:34.573+0000 7f56e7688a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:34.574+0000 7f56e7688a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:34.587 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:34.588 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:34.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:34.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:34.592 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:34.593 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:34.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:34.593 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:34Z" 2024-05-26T21:32:34.593 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:34.594 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.635928_652.1 2024-05-26T21:32:39.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:39.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:39.595 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:39.693+0000 7f51f65aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:39.694+0000 7f51f65aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:39.707 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:39.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:39.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:32:39.710 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:39.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:39.712 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:39Z" 2024-05-26T21:32:39.713 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:39.714 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756068.677571_656.1 2024-05-26T21:32:44.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:44.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:44.716 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:44.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:44.801+0000 7f53571bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:44.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:44.802+0000 7f53571bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:44.816 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.569434_646.1", 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.569434Z", 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:44Z" 2024-05-26T21:32:44.823 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:44.824 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.569434_646.1 2024-05-26T21:32:49.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:49.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:49.825 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:49.881 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:49.881 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:49.917+0000 7f9f4311ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:49.918+0000 7f9f4311ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:49.932 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:49.933 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:49.934 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:49.935 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:49.937 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:49Z" 2024-05-26T21:32:49.938 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:49.938 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756387.702842_717.1 2024-05-26T21:32:54.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:32:54.940 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:32:55.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:32:55.014 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:32:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:55.147+0000 7fa62299fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:32:55.148+0000 7fa62299fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:32:55.161 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:55.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.648499_654.1", 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.648499Z", 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:32:55.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:32:55.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:32:55.167 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:32:55.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:32:55.167 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:32:55Z" 2024-05-26T21:32:55.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:32:55.168 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716758870.157992_924.1 target=1_1716756067.648499_654.1 2024-05-26T21:33:00.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-05-26T21:33:00.170 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-67 2024-05-26T21:33:00.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:00.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:00.219 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:00.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:00.272 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:33:00.320 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:33:00.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:00.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:00.320 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716759180.191311_938.1", 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:33:00.191311Z" 2024-05-26T21:33:00.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:00.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:00.322 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:00.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:33:00.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:00.322 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:33:00.323 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:33:00.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:00.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:00.323 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:00.380 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:00.380 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:00.413 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:33:00.414+0000 7f47081c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:00.414 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:33:00.415+0000 7f47081c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:00.427 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:33:00.427 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:33:00.427 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716758870.157992_924.1", 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:27:50.157992Z", 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:00.431 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:00.432 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:33:00Z" 2024-05-26T21:33:00.433 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:33:00.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716758870.157992_924.1 2024-05-26T21:33:05.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:05.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:05.435 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:05.493 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:05.493 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:05.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:33:05.526+0000 7f76197e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:05.526 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:33:05.527+0000 7f76197e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:33:05.539 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:05.540 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:33:05.541 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716759180.191311_938.1", 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:33:00.191311Z", 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:05.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:05.543 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:33:05Z" 2024-05-26T21:33:05.544 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:33:05.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:33:05.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:33:05.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:05.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:05.546 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:05.633+0000 7f9a86f7fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:05.634+0000 7f9a86f7fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:05.649 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:05.650 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:05.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.632586_650.1", 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.632586Z", 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:05.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:05Z" 2024-05-26T21:33:05.654 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:05.655 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756067.632586_650.1 2024-05-26T21:33:10.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:10.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:10.656 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:10.711 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:10.711 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:10.745+0000 7fe294c88a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:10.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:10.746+0000 7fe294c88a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:10.760 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:10.760 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:10.760 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:10.760 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:10.761 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:10.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756068.677571_656.1", 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:08.677571Z", 2024-05-26T21:33:10.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:10.764 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:10Z" 2024-05-26T21:33:10.765 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:10.766 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756068.677571_656.1 2024-05-26T21:33:10.832 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:33:10.828+0000 7f2531eeb700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:33:10.832 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:33:10.829+0000 7f252bedf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:33:15.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:15.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:15.767 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:15.823 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:15.824 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:15.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:15.856+0000 7fd8ae9c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:15.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:15.857+0000 7fd8ae9c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:15.870 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:15.870 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:15.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:15.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.635928_652.1", 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:15.873 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.635928Z", 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:15.874 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:15.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:15Z" 2024-05-26T21:33:15.876 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:15.877 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756067.635928_652.1 2024-05-26T21:33:20.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:20.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:20.878 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:20.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:20.931 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:20.963+0000 7f20bb658a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:20.964+0000 7f20bb658a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:20.978 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:20.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:20.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:20.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:20.982 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:20.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:20.983 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:20Z" 2024-05-26T21:33:20.983 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:20.984 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756067.572561_648.1 2024-05-26T21:33:25.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:25.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:25.986 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:26.059 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:26.060 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:26.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:26.092+0000 7f57c1b65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:26.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:26.092+0000 7f57c1b65a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:26.106 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756067.572561_648.1", 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:07.572561Z", 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:26Z" 2024-05-26T21:33:26.112 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:26.113 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756067.572561_648.1 2024-05-26T21:33:31.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:31.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:31.115 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:31.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:31.199+0000 7feac5da5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:31.200+0000 7feac5da5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:31.215 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:31.216 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:31.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:31.218 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:31.219 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:31Z" 2024-05-26T21:33:31.220 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:31.221 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.792493_659.1 2024-05-26T21:33:36.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:36.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:36.223 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:36.284 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:36.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:36.322+0000 7fd5c18c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:36.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:36.323+0000 7fd5c18c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:36.340 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:36.340 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:36.340 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:36.340 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:36.340 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:36.341 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:36.344 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:36.345 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:36.346 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:36Z" 2024-05-26T21:33:36.346 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:36.347 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756387.702842_717.1 2024-05-26T21:33:41.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:41.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:41.348 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:41.554+0000 7f58f0c35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:41.555+0000 7f58f0c35a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:41.570 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:41.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.817362_665.1", 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.817362Z", 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:41.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:41.573 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:41Z" 2024-05-26T21:33:41.574 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:41.575 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.817362_665.1 2024-05-26T21:33:46.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:46.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:46.576 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:46.627 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:46.627 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:46.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:46.660+0000 7f42314efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:46.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:46.661+0000 7f42314efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:46.675 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:46.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:46.679 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:46Z" 2024-05-26T21:33:46.679 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:46.680 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:33:51.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:51.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:51.681 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:51.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:51.765+0000 7f68c0c27a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:51.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:51.766+0000 7f68c0c27a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:51.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:51.784 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:51.785 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:51.786 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:51Z" 2024-05-26T21:33:51.788 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:51.789 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.792493_659.1 2024-05-26T21:33:56.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:56.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:33:56.790 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:33:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:33:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:33:56.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:56.886+0000 7f02e81b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:56.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:33:56.887+0000 7f02e81b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:33:56.902 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:56.903 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:56.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.842846_668.1", 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.842846Z", 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.906 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:33:56.907 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:33:56.907 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:33:56.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:33:56.907 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:33:56Z" 2024-05-26T21:33:56.907 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:33:56.908 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.842846_668.1 2024-05-26T21:34:01.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:01.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:01.909 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:02.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:02.014+0000 7ff1b95c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:02.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:02.016+0000 7ff1b95c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:02.030 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:02.030 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:02.030 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:02.030 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:02.031 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:02.032 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:34:02.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:02.034 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:02.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:02Z" 2024-05-26T21:34:02.036 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:02.037 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.800302_662.1 2024-05-26T21:34:07.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:07.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:07.039 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:07.145+0000 7f16f1398a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:07.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:07.149+0000 7f16f1398a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:07Z" 2024-05-26T21:34:07.249 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:07.250 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.800302_662.1 2024-05-26T21:34:12.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:12.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:12.252 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:12.306 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:12.306 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:12.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:12.338+0000 7f88f7c04a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:12.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:12.339+0000 7f88f7c04a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:12.355 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:12.356 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:12.357 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:12.358 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:12.359 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:12.360 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:12.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:12.361 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:12Z" 2024-05-26T21:34:12.361 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:12.361 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.813012_663.1 2024-05-26T21:34:17.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:17.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:17.363 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:17.421 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:17.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:17.458+0000 7f5b5ce1aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:17.459+0000 7f5b5ce1aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:17.473 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:17.474 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:17.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:17.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:17.477 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:17Z" 2024-05-26T21:34:17.478 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:17.479 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.735465_658.1 2024-05-26T21:34:22.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:22.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:22.481 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:22.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:22.572+0000 7fea892c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:22.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:22.573+0000 7fea892c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:22.589 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:22.590 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:22.591 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:22.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:22Z" 2024-05-26T21:34:22.593 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:22.593 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.735465_658.1 2024-05-26T21:34:27.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:27.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:27.595 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:27.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:27.690+0000 7f378c339a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:27.691+0000 7f378c339a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:27.706 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:27.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:27.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.842846_668.1", 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.842846Z", 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:27.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:27Z" 2024-05-26T21:34:27.711 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:27.712 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.842846_668.1 2024-05-26T21:34:32.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:32.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:32.714 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:32.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:32.821+0000 7f1fe245ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:32.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:32.822+0000 7f1fe245ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:32.835 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:32.836 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:32.837 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:32.838 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:32Z" 2024-05-26T21:34:32.840 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:32.841 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:34:37.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:37.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:37.843 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:37.929+0000 7fbe2c2d4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:37.930+0000 7fbe2c2d4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.944 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:37.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:37.947 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:37.947 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:37.947 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:37.947 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:37.949 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:37.949 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:37.949 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:37.950 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:37.951 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:37.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:37.951 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:37Z" 2024-05-26T21:34:37.951 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:37.952 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:34:42.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:42.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:42.954 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:43.013 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:43.013 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:43.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:43.050+0000 7f426a927a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:43.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:43.052+0000 7f426a927a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:43Z" 2024-05-26T21:34:43.073 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:43.074 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:34:48.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:48.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:48.075 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:48.171+0000 7f53d6f66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:48.172+0000 7f53d6f66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:48.187 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:48.189 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:48.190 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:48.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:48Z" 2024-05-26T21:34:48.192 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:48.193 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:34:53.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:53.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:53.195 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:53.260 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:53.260 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:53.391+0000 7f7773dc1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:53.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:53.394+0000 7f7773dc1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:53.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:53.425 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:53.426 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:53.428 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:53.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:53.429 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:53Z" 2024-05-26T21:34:53.429 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:53.430 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.797274_661.1 2024-05-26T21:34:58.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:58.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:34:58.431 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:34:58.483 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:34:58.483 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:34:58.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:58.515+0000 7f8731040a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:58.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:34:58.516+0000 7f8731040a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:34:58.529 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:58.530 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:58.531 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:58.532 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:34:58.533 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:34:58.534 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:34:58Z" 2024-05-26T21:34:58.535 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:34:58.536 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.795957_660.1 2024-05-26T21:35:03.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:03.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:03.538 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:03.622+0000 7f3f8c3cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:03.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:03.622+0000 7f3f8c3cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:03.636 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:03.636 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:03.636 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:03.636 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:03.637 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:03.638 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.842846_668.1", 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:03.639 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.842846Z", 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:03.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:03Z" 2024-05-26T21:35:03.641 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:03.642 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.842846_668.1 2024-05-26T21:35:08.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:08.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:08.644 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:08.701 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:08.701 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:08.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:08.735+0000 7fa55488da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:08.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:08.736+0000 7fa55488da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:08.750 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:08.750 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:08.750 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:08.750 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:08.750 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:08.751 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:08.752 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:08.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:08.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:08Z" 2024-05-26T21:35:08.755 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:08.756 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.800302_662.1 2024-05-26T21:35:13.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:13.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:13.758 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:13.816 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:13.816 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:13.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:13.852+0000 7ff722f91a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:13.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:13.853+0000 7ff722f91a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:13.868 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:13.868 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:13.868 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:13.868 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:13.869 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:13.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:13.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:13Z" 2024-05-26T21:35:13.873 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:13.874 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:35:18.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:18.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:18.875 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:18.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:18.970+0000 7f16c50cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:18.971+0000 7f16c50cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:18.986 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:18.986 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:18.986 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:18.986 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:18.987 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:18.988 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:18.991 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:18.992 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:18Z" 2024-05-26T21:35:18.994 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:18.995 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:35:23.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:23.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:23.996 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:24.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:24.152+0000 7fb69dca5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:24.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:24.153+0000 7fb69dca5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.167 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:24.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:24Z" 2024-05-26T21:35:24.172 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:24.173 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:35:29.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:29.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:29.174 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:29.260+0000 7f8eb5a5ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:29.261+0000 7f8eb5a5ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:29.274 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:29.274 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:29.274 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:29.274 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:29.275 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:29.276 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:35:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:29.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:29Z" 2024-05-26T21:35:29.279 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:29.280 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:35:34.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:34.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:34.282 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:34.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:34.379+0000 7f91c3fbea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:34.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:34.381+0000 7f91c3fbea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:34.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:34.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:34.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:34.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:34.399 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:34Z" 2024-05-26T21:35:34.400 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:34.401 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.813012_663.1 2024-05-26T21:35:39.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:39.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:39.403 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:39.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:39.519+0000 7feb8ab80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:39.522+0000 7feb8ab80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:39.630 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:39.630 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:39.630 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:39.631 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:39.632 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:39.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:39.634 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:39.635 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:39.636 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:39.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:39.636 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:39Z" 2024-05-26T21:35:39.636 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:39.636 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.795957_660.1 2024-05-26T21:35:44.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:44.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:44.637 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:44.725+0000 7f832e487a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:44.726+0000 7f832e487a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:44.739 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:44Z" 2024-05-26T21:35:44.745 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:44.746 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.800302_662.1 2024-05-26T21:35:49.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:49.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:49.747 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:49.842+0000 7f523f5bfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:49.843+0000 7f523f5bfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:49.858 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:49.858 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:49.858 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:49.859 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:49.860 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:49.861 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:49.862 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:49.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:49.864 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:49Z" 2024-05-26T21:35:49.864 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:49.865 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.792493_659.1 2024-05-26T21:35:54.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:54.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:54.867 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:35:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:35:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:35:54.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:54.963+0000 7f893e8f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:54.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:35:54.964+0000 7f893e8f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:35:54.978 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:35:54.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:54.980 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:35:54.981 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.817362_665.1", 2024-05-26T21:35:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.817362Z", 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:35:54.983 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:35:54.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:35:54Z" 2024-05-26T21:35:54.985 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:35:54.987 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.817362_665.1 2024-05-26T21:35:59.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:59.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:35:59.988 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:00.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:00.084+0000 7f6d820cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:00.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:00.086+0000 7f6d820cea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.743853_721.1", 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.743853Z", 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:00Z" 2024-05-26T21:36:00.106 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:00.107 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756387.743853_721.1 2024-05-26T21:36:05.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:05.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:05.108 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:05.186 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:05.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:05.329+0000 7f3353c2da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:05.331+0000 7f3353c2da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:05.344 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:05.345 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:05.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:05.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:05.348 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:05Z" 2024-05-26T21:36:05.349 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:05.350 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.813012_663.1 2024-05-26T21:36:10.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:10.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:10.351 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:10.403 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:10.403 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:10.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:10.435+0000 7f21a9ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:10.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:10.436+0000 7f21a9ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:10.449 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:10.450 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:36:10.451 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:10.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:10Z" 2024-05-26T21:36:10.453 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:10.454 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.795957_660.1 2024-05-26T21:36:15.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:15.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:15.456 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:15.541+0000 7f3499156a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:15.542+0000 7f3499156a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:15.555 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:15.555 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:15.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:15Z" 2024-05-26T21:36:15.560 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:15.561 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:36:20.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:20.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:20.563 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:20.620 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:20.620 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:20.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:20.656+0000 7f4f084e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:20.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:20.657+0000 7f4f084e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:20.671 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:20.671 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:20.672 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:20.674 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:20.675 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:20Z" 2024-05-26T21:36:20.676 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:20.677 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.820209_666.1 2024-05-26T21:36:25.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:25.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:25.679 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:25.773+0000 7f68b7cd3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:25.774+0000 7f68b7cd3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:25.788 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:25.789 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:25.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.555007_818.1", 2024-05-26T21:36:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.555007Z", 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:25.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:25Z" 2024-05-26T21:36:25.793 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:25.794 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716757758.555007_818.1 2024-05-26T21:36:30.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:30.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:30.796 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:30.871 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:30.871 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:31.004+0000 7fd44c641a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:31.010+0000 7fd44c641a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:31.034 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:31.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:31.036 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.846685_670.1", 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:31.037 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.846685Z", 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:31.038 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:31Z" 2024-05-26T21:36:31.039 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:31.040 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.846685_670.1 2024-05-26T21:36:36.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:36.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:36.042 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:36.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:36.128+0000 7fca6ac0fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:36.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:36.129+0000 7fca6ac0fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:36Z" 2024-05-26T21:36:36.147 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:36.148 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.813012_663.1 2024-05-26T21:36:41.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:41.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:41.149 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:41.239+0000 7fc82f1e9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:41.240+0000 7fc82f1e9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:41.255 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:41.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:41.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:41.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:41.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:41.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.260 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:41.260 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:41.260 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:41.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:41.261 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:41Z" 2024-05-26T21:36:41.261 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:41.262 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:36:46.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:46.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:46.264 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:46.323 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:46.323 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:46.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:46.359+0000 7fc0d3d61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:46.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:46.360+0000 7fc0d3d61a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:46.377 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:46.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:46.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:46.380 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:46.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:46.381 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:46Z" 2024-05-26T21:36:46.381 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:46.382 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.735465_658.1 2024-05-26T21:36:51.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:51.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:51.383 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:51.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:51.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:51.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:51.478+0000 7fd37a9d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:51.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:51.479+0000 7fd37a9d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:51.493 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:51.493 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:51.493 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:51.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:51.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:51.496 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:51.497 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:51Z" 2024-05-26T21:36:51.498 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:51.499 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.800302_662.1 2024-05-26T21:36:56.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:56.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:36:56.501 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:36:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:36:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:36:56.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:56.715+0000 7f5b820bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:56.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:36:56.716+0000 7f5b820bca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:36:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.842846_668.1", 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.842846Z", 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:36:56Z" 2024-05-26T21:36:56.734 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:36:56.735 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.842846_668.1 2024-05-26T21:37:01.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:01.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:01.737 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:01.789 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:01.789 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:01.825+0000 7f83c4d0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:01.827+0000 7f83c4d0ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:01.840 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:01.841 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:01.842 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:01.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:01.844 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:01.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:01.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:01.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:01.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:01.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:01.847 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:01.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:01.848 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:01Z" 2024-05-26T21:37:01.848 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:01.849 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:37:06.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:06.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:06.851 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:06.946+0000 7f1e2facca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:06.947+0000 7f1e2facca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:06.961 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:06.962 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:06.963 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:06.964 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:06.965 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:06.966 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:06.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:06.967 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:06Z" 2024-05-26T21:37:06.967 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:06.968 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.735465_658.1 2024-05-26T21:37:11.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:11.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:11.969 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:12.028 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:12.028 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:12.065+0000 7fa8fe2aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:12.066+0000 7fa8fe2aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:12.081 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:12.082 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:12.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:12.085 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:12.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:12Z" 2024-05-26T21:37:12.087 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:12.088 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:37:17.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:17.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:17.089 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:17.148 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:17.148 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:17.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:17.184+0000 7f418c178a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:17.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:17.185+0000 7f418c178a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:17.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:17.201 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:17.203 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:17.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:17.205 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:17Z" 2024-05-26T21:37:17.205 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:17.206 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.839903_667.1 2024-05-26T21:37:22.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:22.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:22.207 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:22.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:22.349+0000 7f984a964a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:22.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:22.350+0000 7f984a964a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:22.363 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:22.363 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:22.364 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:22.365 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:22.366 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:22.367 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.368 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:22.369 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:22.369 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:22.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:22.369 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:22Z" 2024-05-26T21:37:22.369 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:22.370 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.735465_658.1 2024-05-26T21:37:27.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:27.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:27.372 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:27.458+0000 7f358c83da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:27.459+0000 7f358c83da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:27Z" 2024-05-26T21:37:27.477 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:27.478 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.797274_661.1 2024-05-26T21:37:32.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:32.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:32.479 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:32.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:32.566+0000 7fe00362fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:32.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:32.567+0000 7fe00362fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:32.581 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.582 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:32.583 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.702842_717.1", 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.702842Z", 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:32.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:32.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:32Z" 2024-05-26T21:37:32.587 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:32.588 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756387.702842_717.1 2024-05-26T21:37:37.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:37.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:37.589 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:37.648 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:37.648 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:37.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:37.685+0000 7f9ff61eba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:37.686+0000 7f9ff61eba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:37.700 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:37.700 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:37.700 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:37.701 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:37.702 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:37.703 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:37.704 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:37Z" 2024-05-26T21:37:37.705 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:37.706 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.792493_659.1 2024-05-26T21:37:42.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:42.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:42.708 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:42.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:42.803+0000 7f0b0801ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:42.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:42.804+0000 7f0b0801ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:42.818 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.747363_723.1", 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.747363Z", 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:42Z" 2024-05-26T21:37:42.823 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:42.825 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756387.747363_723.1 2024-05-26T21:37:47.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:47.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:47.827 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:47.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:48.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:48.043+0000 7f751c34fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:48.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:48.044+0000 7f751c34fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:48.057 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:48.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:48.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:48.060 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:48.061 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:48.061 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:48.061 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:37:48.061 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:48.061 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:48.062 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:48.063 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:48.063 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:48.063 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:48.063 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:48.063 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:48.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:48Z" 2024-05-26T21:37:48.065 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:48.067 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.792493_659.1 2024-05-26T21:37:53.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:53.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:53.068 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:53.155+0000 7fdfe998ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:53.156+0000 7fdfe998ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:53.169 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:53.169 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:53.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:53.171 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:53.172 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:53.175 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:53Z" 2024-05-26T21:37:53.175 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:53.176 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.797274_661.1 2024-05-26T21:37:58.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:58.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:37:58.177 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:37:58.236 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:37:58.237 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:37:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:58.274+0000 7f8fe5c64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:58.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:37:58.275+0000 7f8fe5c64a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:37:58.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:37:58.291 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:58.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:58.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:37:58.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:37:58Z" 2024-05-26T21:37:58.295 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:37:58.296 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716757758.517177_814.1 2024-05-26T21:38:03.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:03.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:03.297 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:03.356 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:03.357 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:03.393+0000 7f075f1d1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:03.394+0000 7f075f1d1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:03.408 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:03.409 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:03.410 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.769625_725.1", 2024-05-26T21:38:03.411 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.769625Z", 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:03.412 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:03Z" 2024-05-26T21:38:03.413 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:03.415 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756387.769625_725.1 2024-05-26T21:38:08.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:08.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:08.416 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:08.473 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:08.473 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:08.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:08.509+0000 7f9dde408a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:08.510+0000 7f9dde408a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:08.524 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:08.526 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:08.527 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.844053_669.1", 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:08.528 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.844053Z", 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:08.529 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.530 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:08.531 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:08.531 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:08.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:08.531 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:08Z" 2024-05-26T21:38:08.531 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:08.532 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759180.191311_938.1 target=1_1716756069.844053_669.1 2024-05-26T21:38:13.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2024-05-26T21:38:13.534 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-68 2024-05-26T21:38:13.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.691 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:13.774 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:13.774 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:38:13.818 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:38:13.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:13.818 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:13.818 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:38:13.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716759493.557190_946.1", 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:38:13.557190Z" 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:38:13.819 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:13.820 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:38:13.821 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:38:13.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.821 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:13.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:13.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:13.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:38:13.911+0000 7f763633aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:13.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:38:13.912+0000 7f763633aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:13.925 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:13.926 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716759493.557190_946.1", 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:38:13.557190Z", 2024-05-26T21:38:13.928 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:38:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:38:13Z" 2024-05-26T21:38:13.930 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:38:13.931 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:38:13.931 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:38:13.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:13.932 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:13.984 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:13.985 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:14.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:14.017+0000 7ffb76ca1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:14.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:14.018+0000 7ffb76ca1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:14.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:14.033 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:14.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:38:14.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:14.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:14Z" 2024-05-26T21:38:14.037 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:14.039 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.797274_661.1 2024-05-26T21:38:19.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:19.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:19.040 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:19.097 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:19.132+0000 7f3a65ffca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:19.133+0000 7f3a65ffca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:19.147 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:19.148 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:19.149 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:19.150 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:19.151 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:19.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:19Z" 2024-05-26T21:38:19.153 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:19.154 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.795957_660.1 2024-05-26T21:38:21.928 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:38:21.928+0000 7f252a6dc700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-05-26T21:38:24.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:24.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:24.156 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:24.216 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:24.216 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:24.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:24.253+0000 7fc50f0e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:24.253+0000 7fc50f0e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:24.268 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:24.268 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:24.269 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:24.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:24.271 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:24.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:24Z" 2024-05-26T21:38:24.273 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:24.274 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.813012_663.1 2024-05-26T21:38:29.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:29.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:29.276 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:29.334 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:29.335 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:29.372+0000 7f4b33640a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:29.373+0000 7f4b33640a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:29.387 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:29.387 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:29.387 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:29.387 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:29.387 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:29.388 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:29.389 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:29.390 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:29.392 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:38:29.393 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:29.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:29.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:29.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:29.397 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:29.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:29.397 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:29Z" 2024-05-26T21:38:29.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:29.399 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.800302_662.1 2024-05-26T21:38:34.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:34.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:34.401 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:34.567+0000 7ffb0739ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:34.568+0000 7ffb0739ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:34.581 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:34.582 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:34.583 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:34.584 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:34.585 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:34.586 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:34Z" 2024-05-26T21:38:34.587 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:34.587 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.839903_667.1 2024-05-26T21:38:39.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:39.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:39.589 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:39.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:39.673+0000 7f2545d68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:39.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:39.674+0000 7f2545d68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:39.687 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:39.688 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:39.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.797274_661.1", 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.797274Z", 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:39.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:39.691 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:39Z" 2024-05-26T21:38:39.692 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:39.693 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.797274_661.1 2024-05-26T21:38:44.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:44.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:44.694 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:44.751 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:44.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:44.785+0000 7fe085fffa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:44.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:44.786+0000 7fe085fffa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756387.746318_722.1", 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:46:27.746318Z", 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:44Z" 2024-05-26T21:38:44.805 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:44.806 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756387.746318_722.1 2024-05-26T21:38:49.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:49.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:49.807 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:49.866 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:49.866 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:49.903+0000 7fb47db87a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:49.905+0000 7fb47db87a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:49.922 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:49.922 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:49.922 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:49.923 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:49.924 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:49.925 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.816100_664.1", 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.816100Z", 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:49.926 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:49.927 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:49.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:49.928 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:49Z" 2024-05-26T21:38:49.928 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:49.928 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.816100_664.1 2024-05-26T21:38:54.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:54.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:38:54.929 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:38:54.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:38:54.988 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:38:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:55.024+0000 7f761dd5aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:38:55.026+0000 7f761dd5aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:38:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:38:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.813012_663.1", 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.813012Z", 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:38:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:38:55Z" 2024-05-26T21:38:55.047 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:38:55.048 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.813012_663.1 2024-05-26T21:39:00.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:00.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:00.050 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:00.108 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:00.145+0000 7f2914ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:00.146+0000 7f2914ce3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:00.161 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:00.161 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:00.161 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:00.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:00.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:39:00.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:00Z" 2024-05-26T21:39:00.166 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:00.167 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.820209_666.1 2024-05-26T21:39:05.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:05.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:05.168 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:05.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:05.283+0000 7fa80ae99a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:05.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:05.284+0000 7fa80ae99a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:05.298 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:05.298 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:05.299 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:05.300 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:05.301 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.800302_662.1", 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.800302Z", 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:05.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.303 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:05.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:05.304 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:05.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:05.304 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:05Z" 2024-05-26T21:39:05.304 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:05.305 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.800302_662.1 2024-05-26T21:39:10.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:10.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:10.306 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:10.360 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:10.360 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:10.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:10.395+0000 7f2d0f964a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:10.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:10.397+0000 7f2d0f964a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.792493_659.1", 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.792493Z", 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:10.414 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:10Z" 2024-05-26T21:39:10.415 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:10.416 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.792493_659.1 2024-05-26T21:39:15.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:15.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:15.417 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:15.471 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:15.471 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:15.504+0000 7f4bd3289a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:15.505+0000 7f4bd3289a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:15.518 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:15.519 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:15.519 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:15.519 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:15.519 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:15.522 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:15.523 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:15.525 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:15.526 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:15Z" 2024-05-26T21:39:15.527 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:15.528 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.735465_658.1 2024-05-26T21:39:20.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:20.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:20.529 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:20.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:20.617+0000 7f87c0725a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:20.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:20.618+0000 7f87c0725a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:20.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:20.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.735465_658.1", 2024-05-26T21:39:20.634 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.735465Z", 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:20.640 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:20Z" 2024-05-26T21:39:20.641 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:20.642 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.735465_658.1 2024-05-26T21:39:25.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:25.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:25.644 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:25.703 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:25.703 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:25.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:25.741+0000 7f5b87431a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:25.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:25.742+0000 7f5b87431a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:25.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:25.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:25.759 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:25.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:25.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:25.760 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:25.761 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:25.761 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:25.761 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:39:25.761 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:25.762 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:25.763 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:25Z" 2024-05-26T21:39:25.764 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:25.765 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.839903_667.1 2024-05-26T21:39:30.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:30.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:30.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:30.817 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:30.817 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:30.849+0000 7f7c4d92ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:30.850+0000 7f7c4d92ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:30.863 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.820209_666.1", 2024-05-26T21:39:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.820209Z", 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:30Z" 2024-05-26T21:39:30.868 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:30.870 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.820209_666.1 2024-05-26T21:39:35.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:35.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:35.872 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:35.959+0000 7f923ea55a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:35.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:35.960+0000 7f923ea55a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:35.973 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:35.973 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:35.973 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:35.973 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:35.974 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:35.975 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.839903_667.1", 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.839903Z", 2024-05-26T21:39:35.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:35.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:35Z" 2024-05-26T21:39:35.978 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:35.979 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.839903_667.1 2024-05-26T21:39:40.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:40.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:40.980 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:41.038 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:41.038 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:41.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:41.074+0000 7fa23d1d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:41.075+0000 7fa23d1d7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:41.090 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716756069.795957_660.1", 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:41:09.795957Z", 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:41Z" 2024-05-26T21:39:41.096 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:41.097 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716756069.795957_660.1 2024-05-26T21:39:46.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:46.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:46.098 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:46.157 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:46.158 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:46.194+0000 7f058d11ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:46.195+0000 7f058d11ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:46.210 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:46.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:46.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.513073_813.1", 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.513073Z", 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:46.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:46.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:46Z" 2024-05-26T21:39:46.215 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:46.216 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.513073_813.1 2024-05-26T21:39:51.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:51.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:51.218 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:51.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:51.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:51.303+0000 7fc8a338ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:51.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:51.304+0000 7fc8a338ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:51.317 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:51.317 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:51.317 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:51.318 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:51.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.523500_815.1", 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:51.320 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.523500Z", 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:51.321 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:51Z" 2024-05-26T21:39:51.322 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:51.323 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.523500_815.1 2024-05-26T21:39:56.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:56.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:39:56.325 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:39:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:39:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:39:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:56.421+0000 7f1cf7032a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:39:56.422+0000 7f1cf7032a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:39:56.436 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:56.437 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:56.438 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:39:56.439 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:39:56.440 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.441 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:39:56.442 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:39:56.442 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:39:56.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:39:56.442 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:39:56Z" 2024-05-26T21:39:56.442 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:39:56.444 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757760.791474_856.1 2024-05-26T21:40:01.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:01.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:01.445 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:01.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:01.504 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:01.540+0000 7fb0a44eca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:01.541+0000 7fb0a44eca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:01.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:01.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757759.737104_848.1", 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:19.737104Z", 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:01.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:01Z" 2024-05-26T21:40:01.561 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:01.562 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757759.737104_848.1 2024-05-26T21:40:06.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:06.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:06.563 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:06.614 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:06.614 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:06.646+0000 7f1c3bf75a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:06.647+0000 7f1c3bf75a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:06.660 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:06.661 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:06.662 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758247.692744_866.1", 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:17:27.692744Z", 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:06Z" 2024-05-26T21:40:06.665 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:06.665 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716758247.692744_866.1 2024-05-26T21:40:11.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:11.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:11.667 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:11.723 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:11.723 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:11.758+0000 7f814303ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:11.759+0000 7f814303ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:11.774 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:11.774 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:11.774 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:11.774 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:11.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:11.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:11.777 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:11.778 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:11Z" 2024-05-26T21:40:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:11.781 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:40:16.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:16.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:16.783 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:16.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:16.877+0000 7fb9cb7dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:16.878+0000 7fb9cb7dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.513073_813.1", 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.513073Z", 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:16Z" 2024-05-26T21:40:16.897 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:16.898 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.513073_813.1 2024-05-26T21:40:21.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:21.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:21.900 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:21.970 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:21.971 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:22.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:22.014+0000 7f1bc017aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:22.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:22.016+0000 7f1bc017aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:22.031 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:22.031 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:22.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:22.034 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:22.036 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:22Z" 2024-05-26T21:40:22.037 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:22.037 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.517177_814.1 2024-05-26T21:40:27.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:27.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:27.039 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:27.095 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:27.095 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:27.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:27.132+0000 7fd6b6a46a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:27.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:27.133+0000 7fd6b6a46a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.146 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:27.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.553199_817.1", 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.553199Z", 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:27.151 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:27.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:27.151 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:27Z" 2024-05-26T21:40:27.151 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:27.152 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.553199_817.1 2024-05-26T21:40:32.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:32.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:32.153 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:32.212 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:32.212 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:32.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:32.247+0000 7f04e57d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:32.249+0000 7f04e57d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:40:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:32Z" 2024-05-26T21:40:32.268 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:32.269 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:40:37.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:37.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:37.271 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:37.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:37.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:37.363+0000 7fc7a20d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:37.364+0000 7fc7a20d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:37.377 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:37.377 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:37.377 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:37.377 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:37.377 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:37.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:37.379 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.561433_819.1", 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.561433Z", 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:37.380 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:37.381 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:37Z" 2024-05-26T21:40:37.382 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:37.383 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.561433_819.1 2024-05-26T21:40:42.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:42.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:42.385 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:42.441 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:42.441 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:42.475+0000 7f40e506ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:42.476+0000 7f40e506ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:42.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757759.737104_848.1", 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:19.737104Z", 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:42Z" 2024-05-26T21:40:42.496 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:42.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757759.737104_848.1 2024-05-26T21:40:47.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:47.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:47.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:47.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:47.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:47.594+0000 7fa5a15afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:47.595+0000 7fa5a15afa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:47.610 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:47.611 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:47.612 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.513073_813.1", 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.513073Z", 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:47.613 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:47.614 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:47.615 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:47Z" 2024-05-26T21:40:47.615 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:47.616 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.513073_813.1 2024-05-26T21:40:52.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:52.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:52.618 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:52.676 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:52.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:52.812+0000 7f6017b57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:52.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:52.813+0000 7f6017b57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:52.828 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:40:52.833 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:52.834 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:52.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.836 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:52.836 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:52.836 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:52.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:52.836 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:52Z" 2024-05-26T21:40:52.837 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:52.838 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:40:57.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:57.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:40:57.840 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:40:57.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:40:57.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:40:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:57.926+0000 7f3b15437a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:40:57.927+0000 7f3b15437a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:40:57.941 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:40:57.941 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:40:57.941 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:40:57.941 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:40:57.942 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:57.943 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.773901_854.1", 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.773901Z", 2024-05-26T21:40:57.944 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:40:57.945 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:40:57Z" 2024-05-26T21:40:57.946 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:40:57.947 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757760.773901_854.1 2024-05-26T21:41:02.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:02.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:02.949 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:03.045+0000 7fced43b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:03.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:03.046+0000 7fced43b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:03.065 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:03.066 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:03.067 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:03.068 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.665760_832.1", 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.665760Z", 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:03.069 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:03.071 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:03.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:03.071 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:03Z" 2024-05-26T21:41:03.071 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:03.072 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.665760_832.1 2024-05-26T21:41:08.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:08.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:08.073 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:08.169 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:08.169 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:08.238+0000 7f9463425a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:08.239+0000 7f9463425a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:08.252 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:08.254 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.708050_840.1", 2024-05-26T21:41:08.255 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.708050Z", 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:08.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:08Z" 2024-05-26T21:41:08.257 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:08.258 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.708050_840.1 2024-05-26T21:41:13.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:13.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:13.259 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:13.312 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:13.312 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:13.347+0000 7fc28ef2ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:13.348+0000 7fc28ef2ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:13.361 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:13.362 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:13.363 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:13.364 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.639793_829.1", 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.639793Z", 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:13.365 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:13Z" 2024-05-26T21:41:13.367 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:13.368 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.639793_829.1 2024-05-26T21:41:18.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:18.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:18.369 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:18.428 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:18.428 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:18.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:18.464+0000 7f74a5693a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:18.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:18.465+0000 7f74a5693a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:18.479 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:18.479 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:18.479 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:18.480 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:18.481 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758561.336424_898.1", 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:18.482 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:22:41.336424Z", 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:18.483 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:18Z" 2024-05-26T21:41:18.484 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:18.485 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716758561.336424_898.1 2024-05-26T21:41:23.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:23.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:23.487 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:23.585 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:23.585 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:23.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:23.647+0000 7fdcfe366a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:23.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:23.648+0000 7fdcfe366a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:23.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:23.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:23.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:23.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:23Z" 2024-05-26T21:41:23.667 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:23.668 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:41:28.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:28.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:28.669 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:28.725 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:28.725 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:28.758+0000 7f39ac7e0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:28.759+0000 7f39ac7e0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.702343_838.1", 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.702343Z", 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:28Z" 2024-05-26T21:41:28.777 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:28.778 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.702343_838.1 2024-05-26T21:41:33.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:33.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:33.779 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:33.836 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:33.836 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:33.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:33.875+0000 7fbc425e3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:33.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:33.876+0000 7fbc425e3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:33.888 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:33.889 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:33.890 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:33.891 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:33.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:33Z" 2024-05-26T21:41:33.893 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:33.894 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.517177_814.1 2024-05-26T21:41:38.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:38.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:38.896 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:38.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:38.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:38.992+0000 7f2e8ee14a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:38.993+0000 7f2e8ee14a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:39.007 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:39.008 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:39.009 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:39.010 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:39.011 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:39.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:39.013 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:38Z" 2024-05-26T21:41:39.013 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:39.014 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:41:44.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:44.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:44.015 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:44.068 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:44.068 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:44.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:44.101+0000 7fede1077a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:44.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:44.102+0000 7fede1077a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:44.115 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.561433_819.1", 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.561433Z", 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:44Z" 2024-05-26T21:41:44.120 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:44.121 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.561433_819.1 2024-05-26T21:41:49.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:49.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:49.123 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:49.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:49.219+0000 7f4d9a90ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:49.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:49.220+0000 7f4d9a90ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:49.235 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:41:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:49Z" 2024-05-26T21:41:49.240 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:49.241 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:41:54.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:54.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:54.242 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:54.342+0000 7fddc1b8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:54.343+0000 7fddc1b8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:54.357 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:54.358 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:54.359 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.727982_844.1", 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:54.360 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.727982Z", 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:54.361 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:54Z" 2024-05-26T21:41:54.362 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:54.363 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.727982_844.1 2024-05-26T21:41:59.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:59.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:41:59.365 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:41:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:41:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:41:59.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:59.471+0000 7fc461ce0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:41:59.472+0000 7fc461ce0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:41:59.486 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:59.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:59.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:41:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:41:59Z" 2024-05-26T21:41:59.491 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:41:59.492 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.517177_814.1 2024-05-26T21:42:04.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:04.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:04.493 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:04.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:04.579+0000 7eff55738a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:04.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:04.580+0000 7eff55738a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:04.593 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:04.593 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:04.594 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:04.595 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.545901_816.1", 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:04.596 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.545901Z", 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:04.597 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:04Z" 2024-05-26T21:42:04.598 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:04.599 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.545901_816.1 2024-05-26T21:42:09.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:09.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:09.601 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:09.661 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:09.661 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:09.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:09.698+0000 7fa61a494a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:09.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:09.699+0000 7fa61a494a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:09.716 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:09.716 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:09.716 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:09.717 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:42:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:09Z" 2024-05-26T21:42:09.721 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:09.722 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.517177_814.1 2024-05-26T21:42:14.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:14.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:14.724 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:14.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:14.934+0000 7f397614fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:14.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:14.936+0000 7f397614fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:14.949 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:14.949 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:14.950 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:14.951 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:14.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:14.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:14Z" 2024-05-26T21:42:14.954 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:14.955 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:42:19.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:19.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:19.956 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:20.008 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:20.008 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:20.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:20.040+0000 7f643666da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:20.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:20.041+0000 7f643666da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:20.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:20.056 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:20.057 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.553199_817.1", 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.553199Z", 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:20.058 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:20.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:20.060 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:20Z" 2024-05-26T21:42:20.060 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:20.061 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.553199_817.1 2024-05-26T21:42:25.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:25.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:25.062 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:25.120 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:25.120 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:25.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:25.157+0000 7f0770341a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:25.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:25.158+0000 7f0770341a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:25.173 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:25.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.513073_813.1", 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.513073Z", 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:25.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:25.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:25.178 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:25Z" 2024-05-26T21:42:25.178 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:25.179 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.513073_813.1 2024-05-26T21:42:30.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:30.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:30.181 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:30.239 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:30.240 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:30.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:30.276+0000 7f73a4f34a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:30.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:30.277+0000 7f73a4f34a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:30.292 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.523500_815.1", 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:30.295 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.523500Z", 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:30.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:30.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:30.298 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:30Z" 2024-05-26T21:42:30.298 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:30.299 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.523500_815.1 2024-05-26T21:42:35.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:35.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:35.300 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:35.351 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:35.351 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:35.385+0000 7ff5e50b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:35.386+0000 7ff5e50b0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:35.399 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:35.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:35.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:35.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:35.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:35Z" 2024-05-26T21:42:35.404 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:35.405 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:42:40.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:40.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:40.407 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:40.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:40.500+0000 7f0bbaf98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:40.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:40.501+0000 7f0bbaf98a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:40.514 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:40.514 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:40.514 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:40.515 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.555007_818.1", 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.555007Z", 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:40.518 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:40.519 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:40.520 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:40.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:40.520 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:40Z" 2024-05-26T21:42:40.520 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:40.521 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.555007_818.1 2024-05-26T21:42:45.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:45.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:45.522 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:45.580 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:45.580 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:45.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:45.617+0000 7f6f0d3b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:45.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:45.619+0000 7f6f0d3b5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.702343_838.1", 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.702343Z", 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:45Z" 2024-05-26T21:42:45.638 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:45.639 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.702343_838.1 2024-05-26T21:42:50.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:50.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:50.640 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:50.755+0000 7fada4642a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:50.756+0000 7fada4642a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:50.771 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:50.772 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.594831_825.1", 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:50.774 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.594831Z", 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:50.775 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:50.776 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:50Z" 2024-05-26T21:42:50.777 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:50.778 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.594831_825.1 2024-05-26T21:42:55.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:55.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:42:55.779 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:42:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:42:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:42:55.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:55.876+0000 7f88b564da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:55.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:42:55.877+0000 7f88b564da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:42:55.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:55.893 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:42:55.894 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.517177_814.1", 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.517177Z", 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:55.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:55.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:42:55Z" 2024-05-26T21:42:55.897 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:42:55.898 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.517177_814.1 2024-05-26T21:43:00.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:00.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:00.900 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:00.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:00.997+0000 7f93af68fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:00.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:00.998+0000 7f93af68fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:01.013 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:01.014 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:01.015 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:01.016 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:01.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.545901_816.1", 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.545901Z", 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:01.018 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:01.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:01Z" 2024-05-26T21:43:01.020 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:01.022 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.545901_816.1 2024-05-26T21:43:06.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:06.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:06.023 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:06.138+0000 7f0906cdca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:06.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:06.139+0000 7f0906cdca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:06.153 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:06.154 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:06.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.513073_813.1", 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.513073Z", 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:06.158 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:06.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:06.158 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:06Z" 2024-05-26T21:43:06.158 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:06.159 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.513073_813.1 2024-05-26T21:43:11.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:11.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:11.160 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:11.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:11.253+0000 7f55de4c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:11.255+0000 7f55de4c0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:11.272 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:11Z" 2024-05-26T21:43:11.273 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:11.274 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.589013_824.1 2024-05-26T21:43:16.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:16.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:16.276 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:16.336 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:16.336 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:16.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:16.376+0000 7f0221e0ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:16.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:16.377+0000 7f0221e0ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:16.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:16.393 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:16.394 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.596285_826.1", 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.596285Z", 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:16.396 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:16Z" 2024-05-26T21:43:16.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:16.398 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759493.557190_946.1 target=1_1716757758.596285_826.1 2024-05-26T21:43:21.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-05-26T21:43:21.401 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-69 2024-05-26T21:43:21.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.499 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:21.673 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:43:21.686 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:43:21.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.686 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:21.686 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716759801.426338_960.1", 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:43:21.426338Z" 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:43:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.688 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:43:21.688 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:43:21.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.689 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:21.747 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:21.747 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:21.784 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:43:21.785+0000 7f522389ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:21.784 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:43:21.786+0000 7f522389ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:43:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:43:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:43:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716759801.426338_960.1", 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:43:21.426338Z", 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:43:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:43:21Z" 2024-05-26T21:43:21.804 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:43:21.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:43:21.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:43:21.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:21.805 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:21.862 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:21.863 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:21.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:21.898+0000 7f5ce58e2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:21.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:21.898+0000 7f5ce58e2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:21.912 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:21.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:21.914 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.561433_819.1", 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.561433Z", 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:21.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:21.916 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:21Z" 2024-05-26T21:43:21.918 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:21.919 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.561433_819.1 2024-05-26T21:43:26.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:26.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:26.921 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:27.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:27.016+0000 7feea9669a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:27.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:27.017+0000 7feea9669a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:27.031 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:27.031 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:27.031 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:27.032 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.589013_824.1", 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.589013Z", 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:27.035 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:27Z" 2024-05-26T21:43:27.036 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:27.037 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.589013_824.1 2024-05-26T21:43:32.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:32.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:32.039 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:32.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:32.129+0000 7f2871feaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:32.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:32.130+0000 7f2871feaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:32.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716759180.187323_936.1", 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:33:00.187323Z", 2024-05-26T21:43:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:32.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:32Z" 2024-05-26T21:43:32.216 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:32.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716759180.187323_936.1 2024-05-26T21:43:37.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:37.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:37.219 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:37.278 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:37.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:37.315+0000 7f5dfa6fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:37.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:37.316+0000 7f5dfa6fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:37.331 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:37.332 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:37.333 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:37.334 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.821262_860.1", 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.821262Z", 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:37.335 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:37.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:37.337 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:37Z" 2024-05-26T21:43:37.337 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:37.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.821262_860.1 2024-05-26T21:43:42.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:42.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:42.339 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:42.448 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:42.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:42.485+0000 7f4f52cfda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:42.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:42.486+0000 7f4f52cfda80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.821262_860.1", 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.821262Z", 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:42Z" 2024-05-26T21:43:42.505 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:42.506 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.821262_860.1 2024-05-26T21:43:47.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:47.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:47.507 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:47.566 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:47.566 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:47.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:47.604+0000 7fe111778a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:47.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:47.605+0000 7fe111778a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:47.622 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:47.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:47Z" 2024-05-26T21:43:47.624 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:47.625 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:43:52.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:52.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:52.626 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:52.689 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:52.689 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:52.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:52.811+0000 7fc4ebd6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:52.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:52.812+0000 7fc4ebd6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:52.825 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:52.825 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:52.828 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.768907_852.1", 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:52.829 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.768907Z", 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:52.830 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:52.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:52Z" 2024-05-26T21:43:52.832 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:52.833 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.768907_852.1 2024-05-26T21:43:57.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:57.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:43:57.835 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:43:57.894 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:43:57.894 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:43:57.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:57.935+0000 7f36a3074a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:57.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:43:57.936+0000 7f36a3074a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:43:57.950 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:43:57.950 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:43:57.950 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:57.951 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:57.952 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758870.093466_913.1", 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:57.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:27:50.093466Z", 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:43:57.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:43:57Z" 2024-05-26T21:43:57.955 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:43:57.956 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716758870.093466_913.1 2024-05-26T21:44:02.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:02.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:02.957 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:03.138+0000 7f0452a8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:03.145+0000 7f0452a8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:03.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:03.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:03.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758247.692744_866.1", 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:17:27.692744Z", 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:03.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:03.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:03Z" 2024-05-26T21:44:03.169 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:03.170 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716758247.692744_866.1 2024-05-26T21:44:08.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:08.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:08.172 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:08.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:08.270+0000 7ff85a51ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:08.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:08.271+0000 7ff85a51ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:08.285 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:08.286 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:08.287 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:08.288 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.768907_852.1", 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.768907Z", 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:08.289 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:08.290 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:08Z" 2024-05-26T21:44:08.291 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:08.292 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.768907_852.1 2024-05-26T21:44:13.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:13.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:13.293 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:13.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:13.429+0000 7fd1f711ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:13.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:13.431+0000 7fd1f711ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:13.444 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:13.444 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:13.445 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:13.446 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.731601_845.1", 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:13.447 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.731601Z", 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:13.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:13Z" 2024-05-26T21:44:13.449 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:13.450 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.731601_845.1 2024-05-26T21:44:18.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:18.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:18.452 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:18.509 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:18.509 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:18.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:18.543+0000 7f1c3da7ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:18.544+0000 7f1c3da7ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:18.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:18.560 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:18.561 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:18.562 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.772454_853.1", 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.772454Z", 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:18.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:18Z" 2024-05-26T21:44:18.565 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:18.566 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.772454_853.1 2024-05-26T21:44:23.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:23.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:23.568 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:23.626 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:23.626 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:23.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:23.664+0000 7f7bdbd42a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:23.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:23.665+0000 7f7bdbd42a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:23.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:23.681 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:23.682 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:23.683 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:23.684 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:23.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:23.684 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:23Z" 2024-05-26T21:44:23.684 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:23.685 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:44:28.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:28.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:28.686 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:28.742 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:28.742 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:28.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:28.779+0000 7fa8ff0bfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:28.780+0000 7fa8ff0bfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:28.794 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.772454_853.1", 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.772454Z", 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:28Z" 2024-05-26T21:44:28.800 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:28.801 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.772454_853.1 2024-05-26T21:44:33.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:33.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:33.803 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:33.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:33.898+0000 7f62b9c24a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:33.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:33.899+0000 7f62b9c24a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:33.914 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:33.917 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:33.917 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:33.918 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:33.919 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:33.920 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:33.921 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:33Z" 2024-05-26T21:44:33.922 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:33.923 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:44:38.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:38.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:38.924 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:39.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:39.011+0000 7efe7262ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:39.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:39.012+0000 7efe7262ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:39.025 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.768907_852.1", 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.768907Z", 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:39.028 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:39.029 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:39.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:39.030 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:39Z" 2024-05-26T21:44:39.030 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:39.031 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.768907_852.1 2024-05-26T21:44:44.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:44.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:44.032 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:44.091 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:44.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:44.127+0000 7fe53902ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:44.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:44.128+0000 7fe53902ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:44.142 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.772454_853.1", 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.772454Z", 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:44Z" 2024-05-26T21:44:44.147 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:44.148 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.772454_853.1 2024-05-26T21:44:49.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:49.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:49.150 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:49.246 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:49.246 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:49.295+0000 7ffa8b77ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:49.297+0000 7ffa8b77ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:49.310 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:49Z" 2024-05-26T21:44:49.315 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:49.316 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:44:54.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:54.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:54.317 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:54.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:54.417+0000 7f0a38f37a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:54.418+0000 7f0a38f37a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.432 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:54.434 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.768907_852.1", 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.768907Z", 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:54.435 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:54Z" 2024-05-26T21:44:54.437 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:54.438 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.768907_852.1 2024-05-26T21:44:59.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:59.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:44:59.440 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:44:59.529 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:44:59.529 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:44:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:59.565+0000 7f1488356a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:44:59.566+0000 7f1488356a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.579 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:59.581 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757759.737104_848.1", 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:19.737104Z", 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:44:59.582 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:44:59.583 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:44:59.584 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:44:59.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:44:59.584 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:44:59Z" 2024-05-26T21:44:59.584 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:44:59.585 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757759.737104_848.1 2024-05-26T21:45:04.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:04.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:04.586 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:04.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:04.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:04.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:04.679+0000 7f4c04d6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:04.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:04.680+0000 7f4c04d6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:04.695 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:04.696 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:04.698 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:04.700 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:04.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:04.700 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:04Z" 2024-05-26T21:45:04.700 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:04.701 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:45:09.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:09.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:09.702 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:09.761 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:09.761 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:09.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:09.870+0000 7f7f6b0dba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:09.872+0000 7f7f6b0dba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:09.897 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:09.898 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:09.899 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.750519_850.1", 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.750519Z", 2024-05-26T21:45:09.900 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:09Z" 2024-05-26T21:45:09.902 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:09.903 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.750519_850.1 2024-05-26T21:45:14.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:14.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:14.905 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:15.000+0000 7ff2694bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:15.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:15.001+0000 7ff2694bea80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:15.015 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:15.016 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:15.017 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:15.017 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:15.017 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:15.017 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:15.017 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:15.018 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:15.019 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716758561.339186_899.1", 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:22:41.339186Z", 2024-05-26T21:45:15.022 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:15.022 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:15.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:15.022 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:15.023 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.024 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:15.025 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:15.025 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:15.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:15.025 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:15Z" 2024-05-26T21:45:15.025 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:15.027 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716758561.339186_899.1 2024-05-26T21:45:20.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:20.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:20.028 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:20.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:20.087 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:20.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:20.123+0000 7f77add4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:20.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:20.125+0000 7f77add4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:20.139 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:20.139 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:20.139 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:20.139 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:20.139 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:20.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:20.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:20.143 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:20.145 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:20.145 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:20.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:20.145 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:20Z" 2024-05-26T21:45:20.146 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:20.147 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:45:25.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:25.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:25.149 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:25.237+0000 7f5b68fa9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:25.238+0000 7f5b68fa9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:25.252 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:25.253 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:25.253 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:25.253 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:25.253 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:25.253 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:25.254 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:25.255 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:25.255 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:25.255 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:25.255 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:25.256 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.731601_845.1", 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.731601Z", 2024-05-26T21:45:25.260 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:25.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:25.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:25.260 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:25.261 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:25.261 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:25.261 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:25.261 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:25.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:25.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:25.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.263 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:25.263 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:25.264 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:25.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:25.264 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:25Z" 2024-05-26T21:45:25.264 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:25.265 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.731601_845.1 2024-05-26T21:45:30.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:30.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:30.267 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:30.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:30.329 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:30.366+0000 7f641d9d9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:30.367+0000 7f641d9d9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:30.381 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:30.381 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:30.381 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:30.382 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.731601_845.1", 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.731601Z", 2024-05-26T21:45:30.384 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:30.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:30Z" 2024-05-26T21:45:30.386 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:30.387 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.731601_845.1 2024-05-26T21:45:35.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:35.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:35.388 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:35.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:35.476+0000 7ff98060aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:35.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:35.477+0000 7ff98060aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:35.491 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:35.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:35.493 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:35.494 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:35Z" 2024-05-26T21:45:35.495 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:35.496 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:45:40.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:40.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:40.498 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:40.557 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:40.557 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:40.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:40.594+0000 7fba6befaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:40.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:40.595+0000 7fba6befaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.754702_851.1", 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.754702Z", 2024-05-26T21:45:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:40Z" 2024-05-26T21:45:40.617 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:40.618 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.754702_851.1 2024-05-26T21:45:45.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:45.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:45.620 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:45.672 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:45.673 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:45.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:45.705+0000 7fccbab1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:45.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:45.706+0000 7fccbab1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.824590_861.1", 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.824590Z", 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:45Z" 2024-05-26T21:45:45.728 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:45.729 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.824590_861.1 2024-05-26T21:45:50.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:50.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:50.731 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:50.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:50.829+0000 7fa8ecd39a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:50.830+0000 7fa8ecd39a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:50.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:50.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:50.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:50.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:50.850 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:50Z" 2024-05-26T21:45:50.851 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:50.851 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:45:55.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:55.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:45:55.853 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:45:55.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:45:55.929 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:45:55.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:55.961+0000 7fed2a081a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:55.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:45:55.962+0000 7fed2a081a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:45:55.975 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:45:55.975 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:45:55.975 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:45:55.975 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:45:55.975 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:45:55.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:55.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.777061_855.1", 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:55.978 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.777061Z", 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:45:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:45:55Z" 2024-05-26T21:45:55.980 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:45:55.981 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.777061_855.1 2024-05-26T21:46:00.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:00.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:00.982 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:01.041 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:01.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:01.078+0000 7feadfc07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:01.079+0000 7feadfc07a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:01.092 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:01.093 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:01.094 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.772454_853.1", 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:01.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.772454Z", 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:01.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:01Z" 2024-05-26T21:46:01.097 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:01.098 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.772454_853.1 2024-05-26T21:46:06.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:06.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:06.100 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:06.158 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:06.158 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:06.195+0000 7f6b1f520a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:06.196+0000 7f6b1f520a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:06.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:06.212 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757760.791474_856.1", 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:20.791474Z", 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:06.214 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:06.215 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:06.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:06.216 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:06Z" 2024-05-26T21:46:06.216 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:06.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757760.791474_856.1 2024-05-26T21:46:11.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:11.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:11.218 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:11.312+0000 7f0fe4abaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:11.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:11.313+0000 7f0fe4abaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:11.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:11.329 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:11.330 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757759.737104_848.1", 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:19.737104Z", 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:11.331 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:11.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:11.333 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:11Z" 2024-05-26T21:46:11.333 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:11.334 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757759.737104_848.1 2024-05-26T21:46:16.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:16.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:16.335 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:16.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:16.432+0000 7f007272da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:16.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:16.433+0000 7f007272da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.448 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:16.450 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.731601_845.1", 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.731601Z", 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:16.451 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:16.452 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:16.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:16.453 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:16Z" 2024-05-26T21:46:16.453 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:16.454 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.731601_845.1 2024-05-26T21:46:21.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:21.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:21.455 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:21.542+0000 7f24b70e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:21.543+0000 7f24b70e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:21.556 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757759.741319_849.1", 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:19.741319Z", 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:21Z" 2024-05-26T21:46:21.561 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:21.562 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757759.741319_849.1 2024-05-26T21:46:26.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:26.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:26.564 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:26.624 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:26.624 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:26.661+0000 7f9d69bdba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:26.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:26.662+0000 7f9d69bdba80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:26.676 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:26.677 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:26.678 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716757758.731601_845.1", 2024-05-26T21:46:26.679 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:09:18.731601Z", 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:26.680 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:26Z" 2024-05-26T21:46:26.681 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:26.682 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716759801.426338_960.1 target=1_1716757758.731601_845.1 2024-05-26T21:46:31.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:31.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:46:31.684 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:46:31.738 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:31.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:31.772+0000 7ff4106aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:31.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:31.773+0000 7ff4106aca80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:46:31.786 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:46:31.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:31.788 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:46:31.789 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716759801.426338_960.1", 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:43:21.426338Z", 2024-05-26T21:46:31.790 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:46:31.791 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:46:31.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:46:31.793 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:46:31Z" 2024-05-26T21:46:31.793 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:46:31.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T21:46:31.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:31.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:31.795 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T21:46:31.860 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:31.861 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:31.893 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-69 2024-05-26T21:46:31.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:31.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:31.908 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:46:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:33.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:33.007+0000 7f0f219b8a80 1 bucket sync caught up with source: 2024-05-26T21:46:33.007 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.48.6, , , , 00000000001.67.6, , , 00000000001.32.6, , 00000000001.32.6] 2024-05-26T21:46:33.007 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.48.6, , , , 00000000001.67.6, , , 00000000001.32.6, , 00000000001.32.6] 2024-05-26T21:46:33.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:33.007+0000 7f0f219b8a80 0 bucket checkpoint complete 2024-05-26T21:46:33.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:33.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:46:33.027 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:46:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:46:33.101 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:46:33.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:33.141+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:46:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:38.141+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:46:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:43.142+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:46:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:48.143+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:46:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:53.144+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:46:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:46:58.144+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:03.145+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:08.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:08.146+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:13.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:13.146+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:18.146+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:23.147+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:28.148+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:33.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:33.148+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:38.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:38.149+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:43.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:43.149+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:48.150+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:53.150+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:47:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:47:58.151+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:03.151+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:08.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:08.152+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:13.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:13.153+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:18.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:18.153+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:23.153+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:28.154+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:33.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:33.154+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:38.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:38.155+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:43.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:43.155+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:48.156+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:53.157+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:48:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:48:58.157+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:03.158+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:08.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:08.159+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:13.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:13.160+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:18.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:18.160+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:23.160+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:28.161+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:33.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:33.161+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:38.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:38.161+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:43.162+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:48.162+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:53.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:53.163+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:49:58.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:49:58.164+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:03.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:03.164+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:08.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:08.165+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:13.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:13.165+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:18.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:18.166+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:23.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:23.167+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:28.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:28.168+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:33.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:33.168+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:38.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:38.168+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:43.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:43.169+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:48.169+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:53.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:53.170+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:50:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:50:58.171+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:03.171+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:08.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:08.172+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:13.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:13.173+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:18.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:18.173+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:23.174+0000 7fc298685a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.175+0000 7fc298685a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T21:51:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.175+0000 7fc298685a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T21:51:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.175+0000 7fc298685a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T21:51:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T21:51:28.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-05-26T21:51:28.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.218 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:51:28.275 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:28.275 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:28.313 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760288.216634_975.1", 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:51:28.216634Z" 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.329 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:51:28.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:51:28.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.331 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:51:28.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:28.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:28.418 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:51:28.418+0000 7f3df841da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:51:28.418 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:51:28.419+0000 7f3df841da80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:51:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:51:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716760288.216634_975.1", 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:51:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:51:28.216634Z", 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:51:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:51:28Z" 2024-05-26T21:51:28.436 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:51:28.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:51:28.437 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:51:28.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:51:28.438 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:51:28.488 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:28.489 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:28.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.520+0000 7fbc3e5a3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:51:28.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.521+0000 7fbc3e5a3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:51:28.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:51:28.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760288.216634_975.1", 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:51:28.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:51:28.216634Z", 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:51:28.538 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:51:28Z" 2024-05-26T21:51:28.539 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:51:28.540 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T21:51:28.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:28.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:28.614 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T21:51:28.672 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:28.672 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:28.704 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-70 2024-05-26T21:51:28.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:28.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:28.717 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:51:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:28.814+0000 7efbf63a6a80 1 waiting to reach incremental sync.. 2024-05-26T21:51:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:33.816+0000 7efbf63a6a80 1 bucket sync caught up with source: 2024-05-26T21:51:33.816 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, 00000000001.49.6, , , , 00000000001.70.6, , , 00000000001.57.6, , 00000000001.42.6] 2024-05-26T21:51:33.816 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, 00000000001.49.6, , , , 00000000001.70.6, , , 00000000001.57.6, , 00000000001.42.6] 2024-05-26T21:51:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:33.816+0000 7efbf63a6a80 0 bucket checkpoint complete 2024-05-26T21:51:33.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:33.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:51:33.837 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:51:33.900 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:51:33.900 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:51:33.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:33.941+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:51:38.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:38.941+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:51:43.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:43.941+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:51:48.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:48.942+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:51:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:53.942+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:51:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:51:58.943+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:03.944+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:08.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:08.944+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:13.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:13.945+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:18.946+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:23.946+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:28.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:28.947+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:33.948+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:38.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:38.949+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:43.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:43.949+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:48.949+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:53.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:53.950+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:52:58.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:52:58.950+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:03.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:03.951+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:08.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:08.951+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:13.952+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:18.952+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:23.953+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:28.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:28.954+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:33.955+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:38.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:38.955+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:43.955+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:48.956+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:53.957+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:53:58.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:53:58.957+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:03.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:03.958+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:08.958+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:13.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:13.959+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:18.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:18.960+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:23.960+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:28.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:28.960+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:33.961+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:38.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:38.962+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:43.962+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:48.963+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:53.964+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:54:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:54:58.964+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:03.964+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:08.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:08.965+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:13.965+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:18.966+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:23.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:23.967+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:28.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:28.967+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:33.968+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:38.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:38.968+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:43.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:43.969+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:48.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:48.969+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:53.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:53.970+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:55:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:55:58.971+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:03.971+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:08.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:08.971+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:13.972+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:18.973+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:23.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:23.973+0000 7f8d2d86fa80 1 waiting to reach incremental sync.. 2024-05-26T21:56:28.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:28.974+0000 7f8d2d86fa80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T21:56:28.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:28.974+0000 7f8d2d86fa80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T21:56:28.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:28.974+0000 7f8d2d86fa80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T21:56:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T21:56:28.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-05-26T21:56:29.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:29.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:29.019 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:56:29.078 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:29.079 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T21:56:29.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760589.017004_981.1", 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:56:29.017004Z" 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:29.129 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T21:56:29.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T21:56:29.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:29.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:29.131 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:56:29.188 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:29.188 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:29.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:56:29.221+0000 7f7935443a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:29.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:56:29.222+0000 7f7935443a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.234 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:29.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716760589.008778_978.1", 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:56:29.008778Z", 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:56:29.238 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:56:29Z" 2024-05-26T21:56:29.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:56:29.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716760589.017004_981.1 target=1_1716760589.008778_978.1 2024-05-26T21:56:34.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:34.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:34.241 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:56:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:56:34.331+0000 7fa951816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:34.331 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T21:56:34.332+0000 7fa951816a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:34.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:34.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:34.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716760589.017004_981.1", 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:56:29.017004Z", 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T21:56:34.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T21:56:34Z" 2024-05-26T21:56:34.350 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T21:56:34.351 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T21:56:34.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T21:56:34.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:34.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T21:56:34.351 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T21:56:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:34.407 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:34.442+0000 7f07a2c09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:34.443+0000 7f07a2c09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T21:56:34.457 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T21:56:34.457 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T21:56:34.457 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:56:34.458 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:56:34.459 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760589.017004_981.1", 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:56:34.460 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:56:29.017004Z", 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T21:56:34.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T21:56:34Z" 2024-05-26T21:56:34.462 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T21:56:34.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T21:56:34.463 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-05-26T21:56:34.464 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T21:56:34.464 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T21:56:34.465+0000 7f1e7c190700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 49200) UID: 0 2024-05-26T21:56:34.464 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T21:56:34.465+0000 7f1e803fda80 -1 shutting down 2024-05-26T21:56:39.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-05-26T21:56:40.565 INFO:tasks.rgw.c2.client.0:Stopped 2024-05-26T21:56:40.565 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-05-26T21:56:40.565 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T21:56:40.566 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:56:40.567+0000 7f253a6fc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 50801) UID: 0 2024-05-26T21:56:40.567 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T21:56:40.567+0000 7f253e969a80 -1 shutting down 2024-05-26T21:56:45.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-05-26T21:56:46.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-05-26T21:56:46.667 INFO:tasks.rgw.c1.client.1:Stopped 2024-05-26T21:56:46.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'xmqxus-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:56:46.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'xmqxus-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T21:56:46.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket xmqxus-71 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T21:56:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:46.847 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T21:56:46.847 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-71 2024-05-26T21:56:46.851 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 4 2024-05-26T21:56:46.956 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-05-26T21:56:46.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-05-26T21:56:46.959 INFO:tasks.rgw.c2.client.0:Started 2024-05-26T21:56:46.960 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T21:56:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T21:56:46.986 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T21:56:46.986 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T21:56:46.986 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T21:56:47.987 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T21:56:48.010 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T21:56:48.011 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T21:56:48.011 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T21:56:48.011 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T21:56:48.012 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-05-26T21:56:48.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-05-26T21:56:48.014 INFO:tasks.rgw.c1.client.1:Started 2024-05-26T21:56:48.015 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8001 2024-05-26T21:56:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T21:56:48.081 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T21:56:48.081 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T21:56:48.081 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi007.front.sepia.ceph.com port 8001: Connection refused 2024-05-26T21:56:49.082 DEBUG:teuthology.orchestra.run.smithi007:> curl http://smithi007.front.sepia.ceph.com:8001 2024-05-26T21:56:49.105 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T21:56:49.105 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2024-05-26T21:56:49.106 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T21:56:49.107 INFO:teuthology.orchestra.run.smithi007.stdout:anonymous 2024-05-26T21:56:49.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:56:49.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:56:49.107 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T21:56:49.174 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:49.174 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:49.207 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-71 2024-05-26T21:56:49.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:56:49.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:56:49.219 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:56:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:56:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:56:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:49.318+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:56:52.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-05-26T21:56:54.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:54.319+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:56:59.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:56:59.320+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:04.321+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:09.321+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:14.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:14.321+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:19.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:19.322+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:24.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:24.322+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:29.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:29.323+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:34.324+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:39.324+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:44.325+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:49.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:49.325+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:54.326+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:57:59.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:57:59.327+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:04.327+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:09.328+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:14.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:14.328+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:19.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:19.329+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:24.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:24.330+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:29.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:29.331+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:34.331+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:39.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:39.332+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:44.332+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:49.333+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:54.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:54.333+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:58:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:58:59.334+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:04.334+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:09.336+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:14.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:14.336+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:19.337+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:24.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:24.337+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:29.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:29.338+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:34.339+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:39.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:39.339+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:44.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:44.340+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:49.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:49.340+0000 7f83e9e87a80 1 waiting to reach incremental sync.. 2024-05-26T21:59:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:54.342+0000 7f83e9e87a80 1 bucket sync caught up with source: 2024-05-26T21:59:54.342 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [00000000003.52.6, 00000000002.57.6, 00000000002.74.6] 2024-05-26T21:59:54.342 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [00000000003.52.6, 00000000002.57.6, 00000000002.74.6] 2024-05-26T21:59:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:54.342+0000 7f83e9e87a80 0 bucket checkpoint complete 2024-05-26T21:59:54.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:59:54.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T21:59:54.363 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T21:59:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T21:59:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T21:59:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:54.470+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T21:59:59.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T21:59:59.470+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:04.471+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:09.471+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:14.473+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:19.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:19.473+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:24.473+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:29.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:29.474+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:34.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:34.474+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:39.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:39.475+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:44.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:44.476+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:49.476+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:54.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:54.477+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:00:59.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:00:59.478+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:04.478+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:09.479+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:14.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:14.480+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:19.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:19.480+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:24.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:24.481+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:29.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:29.481+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:34.482+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:39.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:39.482+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:44.483+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:49.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:49.484+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:54.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:54.484+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:01:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:01:59.485+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:04.485+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:09.486+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:14.487+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:19.488+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:24.488+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:29.488+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:34.489+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:39.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:39.489+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:44.490+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:49.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:49.490+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:54.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:54.490+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:02:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:02:59.491+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:04.491+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:09.492+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:14.492+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:19.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:19.493+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:24.494+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:29.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:29.494+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:34.494+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:39.495+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:44.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:44.495+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:49.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:49.496+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:54.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:54.497+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:03:59.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:03:59.498+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:04.498+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:09.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:09.499+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:14.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:14.500+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:19.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:19.500+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:24.501+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:29.502+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:34.502+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:39.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:39.502+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:44.503+0000 7f39eb6fca80 1 waiting to reach incremental sync.. 2024-05-26T22:04:49.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:49.503+0000 7f39eb6fca80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:04:49.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:49.503+0000 7f39eb6fca80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:04:49.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:49.504+0000 7f39eb6fca80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:04:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:04:49.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2024-05-26T22:04:49.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.525 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:49.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:49.583 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:49.619 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:04:49.632 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:04:49.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.632 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760606.857696_987.1", 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T21:56:46.857696Z" 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T22:04:49.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.634 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:04:49.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:04:49.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.635 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:49.693 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:49.694 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:49.725 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:04:49.726+0000 7f328bbf1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:49.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:04:49.727+0000 7f328bbf1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:49.739 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:49.740 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:49.741 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716760606.857696_987.1", 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:49.742 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T21:56:46.857696Z", 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:04:49Z" 2024-05-26T22:04:49.744 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:04:49.745 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:04:49.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:04:49.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.746 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:49.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:49.830+0000 7fc3aac9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:49.831+0000 7fc3aac9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:49.845 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:04:49.845 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:04:49.845 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:04:49.845 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:04:49.845 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716760606.857696_987.1", 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T21:56:46.857696Z", 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:04:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:04:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:04:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:04:49Z" 2024-05-26T22:04:49.850 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:04:49.851 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:04:49.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-05-26T22:04:49.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.890 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761089.889270_1007.1", 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:04:49.889270Z" 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:04:49.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:04:49.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:49.988 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:50.039 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:50.039 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:50.068 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:04:50.069+0000 7fdb36402a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:50.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:04:50.070+0000 7fdb36402a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761089.889270_1007.1", 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:04:49.889270Z", 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:04:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:04:50Z" 2024-05-26T22:04:50.086 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:04:50.087 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:04:50.087 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:04:50.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:50.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:50.087 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:50.138 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:50.138 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:50.169+0000 7f6731253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:50.170+0000 7f6731253a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:50.184 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:04:50.186 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761089.883098_1004.1", 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:04:49.883098Z", 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:04:50.187 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:04:50.188 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:04:50.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:50.189 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:04:50Z" 2024-05-26T22:04:50.189 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:04:50.190 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716761089.889270_1007.1 target=1_1716761089.883098_1004.1 2024-05-26T22:04:55.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:55.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:55.191 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:55.249 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:55.250 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:55.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:55.285+0000 7f1f14d7ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:55.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:55.286+0000 7f1f14d7ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:04:55.301 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:04:55.301 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:55.302 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:55.303 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:55.304 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761089.889270_1007.1", 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:04:49.889270Z", 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:04:55.305 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:04:55.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:04:55.307 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:04:55Z" 2024-05-26T22:04:55.307 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:04:55.307 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:04:55.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:55.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:04:55.308 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:04:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "objs": [ 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000001", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000005", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000012", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000004", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000011", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000000", 2024-05-26T22:04:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000010", 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000003", 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000013", 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000002", 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout: "obj00000000000000000006" 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:04:55.404 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:04:55.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:04:55.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:04:55.631 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:04:55.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:55.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:55.729 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-72 2024-05-26T22:04:55.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:04:55.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:04:55.743 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:04:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:04:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:04:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:04:55.839+0000 7fdc9b7f7a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:00.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:00.840+0000 7fdc9b7f7a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:05.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:05.841+0000 7fdc9b7f7a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:10.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:10.843+0000 7fdc9b7f7a80 1 bucket sync caught up with source: 2024-05-26T22:05:10.843 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [00000000001.25.6, , 00000000001.38.6, 00000000001.36.6, 00000000001.97.6, 00000000001.30.6, 00000000001.76.6, 00000000001.55.6, 00000000001.60.6, 00000000001.42.6, 00000000001.36.6] 2024-05-26T22:05:10.843 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [00000000001.25.6, , 00000000001.38.6, 00000000001.36.6, 00000000001.97.6, 00000000001.30.6, 00000000001.76.6, 00000000001.55.6, 00000000001.60.6, 00000000001.42.6, 00000000001.36.6] 2024-05-26T22:05:10.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:10.843+0000 7fdc9b7f7a80 0 bucket checkpoint complete 2024-05-26T22:05:10.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:05:10.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:05:10.863 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:05:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:05:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:05:10.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:10.966+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:15.967+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:20.967+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:25.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:25.968+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:30.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:30.969+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:35.969+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:40.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:40.969+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:45.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:45.970+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:50.971+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:05:55.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:05:55.971+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:00.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:00.972+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:05.973+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:10.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:10.973+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:15.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:15.974+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:20.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:20.974+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:25.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:25.975+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:30.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:30.976+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:35.977+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:40.977+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:45.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:45.978+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:50.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:50.978+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:06:55.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:06:55.979+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:00.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:00.979+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:05.980+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:10.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:10.981+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:15.981+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:20.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:20.982+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:25.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:25.982+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:30.983+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:35.983+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:40.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:40.984+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:45.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:45.985+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:50.985+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:07:55.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:07:55.986+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:00.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:00.987+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:05.987+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:10.988+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:15.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:15.989+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:20.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:20.989+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:25.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:25.990+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:30.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:30.991+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:35.991+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:40.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:40.991+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:45.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:45.992+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:50.993+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:08:55.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:08:55.993+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:00.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:00.994+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:05.994+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:10.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:10.995+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:15.995+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:20.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:20.996+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:25.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:25.996+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:30.997+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:35.998+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:40.999+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:45.999+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:51.000+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:09:56.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:09:56.000+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:10:01.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:01.001+0000 7fa057748a80 1 waiting to reach incremental sync.. 2024-05-26T22:10:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.001+0000 7fa057748a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:10:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.001+0000 7fa057748a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:10:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.001+0000 7fa057748a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:10:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:10:06.020 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2024-05-26T22:10:06.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.045 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:10:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:06.142 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:10:06.156 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:10:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.156 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:20:42.623606Z" 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "0.000000" 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:10:06.042849Z" 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:10:06.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T20:22:52.511255Z" 2024-05-26T22:10:06.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.158 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:10:06.159 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:10:06.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.159 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:10:06.215 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:06.215 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:06.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:10:06.246+0000 7f705870ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:10:06.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:10:06.247+0000 7f705870ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:10:06.261 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:10:06.261 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:10:06.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:10:06.263 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-05-26T22:10:06.264 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:10:06.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:10:06.266 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:10:06.042849Z", 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:10:06.267 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:10:06.268 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:10:06Z" 2024-05-26T22:10:06.269 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:10:06.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:10:06.271 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:10:06.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:10:06.272 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:10:06.324 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:06.324 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:06.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.356+0000 7fa240dfda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:10:06.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.357+0000 7fa240dfda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:10:06.370 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:10:06.371 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754842.623606_31.1", 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:20:42.623606Z", 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "", 2024-05-26T22:10:06.372 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "0.000000", 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:10:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:10:06.042849Z", 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716754972.511255_82.1", 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:10:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T20:22:52.511255Z", 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:10:06Z" 2024-05-26T22:10:06.375 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:10:06.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:10:06.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:06.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:06.491 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:10:06.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:06.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:06.580 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-73 2024-05-26T22:10:06.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:06.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:06.595 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:10:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:06.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:06.691+0000 7fdbad9b9a80 1 waiting to reach incremental sync.. 2024-05-26T22:10:11.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:11.693+0000 7fdbad9b9a80 1 bucket sync caught up with source: 2024-05-26T22:10:11.693 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.48.6, 00000000002.58.6, 00000000002.102.6] 2024-05-26T22:10:11.694 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.48.6, 00000000002.58.6, 00000000002.102.6] 2024-05-26T22:10:11.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:11.693+0000 7fdbad9b9a80 0 bucket checkpoint complete 2024-05-26T22:10:11.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:11.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:10:11.713 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:10:11.778 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:10:11.778 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:10:11.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:11.817+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:16.817+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:21.818+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:26.819+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:31.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:31.819+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:36.820+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:41.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:41.820+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:46.821+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:51.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:51.822+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:10:56.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:10:56.823+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:01.823+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:06.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:06.823+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:11.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:11.824+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:16.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:16.825+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:21.825+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:26.826+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:31.827+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:36.827+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:41.828+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:46.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:46.828+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:51.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:51.829+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:11:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:11:56.829+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:01.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:01.830+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:06.830+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:11.831+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:16.832+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:21.832+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:26.833+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:31.833+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:36.834+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:41.835+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:46.835+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:51.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:51.836+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:12:56.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:12:56.837+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:01.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:01.837+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:06.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:06.838+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:11.838+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:16.839+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:21.839+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:26.840+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:31.840+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:36.841+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:41.842+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:46.842+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:51.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:51.843+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:13:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:13:56.844+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:01.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:01.844+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:06.845+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:11.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:11.846+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:16.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:16.846+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:21.847+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:26.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:26.847+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:31.848+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:36.848+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:41.848+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:46.849+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:51.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:51.849+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:14:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:14:56.850+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:15:01.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:01.851+0000 7f1c2d1daa80 1 waiting to reach incremental sync.. 2024-05-26T22:15:06.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:06.851+0000 7f1c2d1daa80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:15:06.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:06.851+0000 7f1c2d1daa80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:15:06.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:06.851+0000 7f1c2d1daa80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:15:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:15:06.871 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2024-05-26T22:15:06.872 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-xmqxus-1 2024-05-26T22:15:06.888 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-xmqxus-2 2024-05-26T22:15:06.911 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-xmqxus-3 2024-05-26T22:15:07.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:07.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:07.936 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.027 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:15:08.041 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:15:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:15:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:10:06.042849Z" 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:15:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.043 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:15:08.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:15:08.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.044 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:08.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:08.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.134 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:08.135+0000 7f46e8a89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.135 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:08.136+0000 7f46e8a89a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.148 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:15:08.148 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.149 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.150 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:15:08.151 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:10:06.042849Z", 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.152 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:15:08Z" 2024-05-26T22:15:08.153 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:15:08.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:15:08.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:15:08.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.155 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:08.244+0000 7fde71c47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:08.245+0000 7fde71c47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.261 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:15:08.262 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761406.042849_1025.1", 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.264 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:10:06.042849Z", 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:15:08.265 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:15:08Z" 2024-05-26T22:15:08.267 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:15:08.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:15:08.269 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-1 zones={test-zone2, test-zone1} 2024-05-26T22:15:08.274 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-1 2024-05-26T22:15:08.274 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-1 zones={test-zone2, test-zone1} 2024-05-26T22:15:08.274 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-1 zones={test-zone3, test-zone1} 2024-05-26T22:15:08.278 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-1 2024-05-26T22:15:08.278 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-1 zones={test-zone3, test-zone1} 2024-05-26T22:15:08.278 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-2 zones={test-zone1, test-zone2} 2024-05-26T22:15:08.282 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-2 2024-05-26T22:15:08.282 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-2 zones={test-zone1, test-zone2} 2024-05-26T22:15:08.282 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-2 zones={test-zone3, test-zone2} 2024-05-26T22:15:08.286 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-2 2024-05-26T22:15:08.286 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-2 zones={test-zone3, test-zone2} 2024-05-26T22:15:08.286 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-3 zones={test-zone1, test-zone3} 2024-05-26T22:15:08.290 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-3 2024-05-26T22:15:08.290 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-3 zones={test-zone1, test-zone3} 2024-05-26T22:15:08.290 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xmqxus-3 zones={test-zone2, test-zone3} 2024-05-26T22:15:08.294 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xmqxus-3 2024-05-26T22:15:08.294 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xmqxus-3 zones={test-zone2, test-zone3} 2024-05-26T22:15:08.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-05-26T22:15:08.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.313 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:15:08.407 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761708.311773_1031.1", 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:08.311773Z" 2024-05-26T22:15:08.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.409 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.409 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:15:08.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.409 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:15:08.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:15:08.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.410 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:08.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:08.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.498 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:08.499+0000 7f9329e2aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:08.500+0000 7f9329e2aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761708.311773_1031.1", 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:08.311773Z", 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:15:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:15:08Z" 2024-05-26T22:15:08.517 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:15:08.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:15:08.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:15:08.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:15:08.519 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:15:08.569 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:08.569 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:08.600+0000 7f54da55ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:08.601+0000 7f54da55ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:15:08.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.616 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:15:08.617 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761708.311773_1031.1", 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:08.311773Z", 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:15:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:15:08.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:15:08.621 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:15:08Z" 2024-05-26T22:15:08.621 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:15:08.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:15:08.625 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-05-26T22:15:08.625 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T22:15:08.626 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T22:15:08.624+0000 7fa6ca61c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 69488) UID: 0 2024-05-26T22:15:08.626 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T22:15:08.624+0000 7fa6ce889a80 -1 shutting down 2024-05-26T22:15:12.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-05-26T22:15:14.727 INFO:tasks.rgw.c2.client.0:Stopped 2024-05-26T22:15:14.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:15:14.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:15:14.807 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T22:15:14.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:14.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:14.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:14.901+0000 7fd4ec509a80 5 RGW-SYNC:data: waiting on data sync status lease 2024-05-26T22:15:14.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:15:14.902+0000 7fd4ec509a80 5 RGW-SYNC:data: acquired data sync status lease 2024-05-26T22:15:14.925 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-05-26T22:15:14.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-05-26T22:15:14.927 INFO:tasks.rgw.c2.client.0:Started 2024-05-26T22:15:14.927 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T22:15:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T22:15:14.949 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T22:15:14.949 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T22:15:14.949 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T22:15:15.950 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T22:15:15.971 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T22:15:15.972 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T22:15:15.972 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T22:15:15.973 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T22:15:15.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:15.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:15.974 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:15:16.068 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:16.069 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:16.102 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-74 2024-05-26T22:15:16.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:16.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:16.115 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:15:16.181 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:16.181 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:16.222+0000 7f81c3f5ea80 1 waiting to reach incremental sync.. 2024-05-26T22:15:17.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-05-26T22:15:21.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:21.222+0000 7f81c3f5ea80 1 waiting to reach incremental sync.. 2024-05-26T22:15:26.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:26.223+0000 7f81c3f5ea80 1 waiting to reach incremental sync.. 2024-05-26T22:15:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:31.225+0000 7f81c3f5ea80 1 bucket sync caught up with source: 2024-05-26T22:15:31.225 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.61.6, 00000000002.30.6, 00000000002.63.6] 2024-05-26T22:15:31.225 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.61.6, 00000000002.30.6, 00000000002.63.6] 2024-05-26T22:15:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:31.225+0000 7f81c3f5ea80 0 bucket checkpoint complete 2024-05-26T22:15:31.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:31.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:15:31.245 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:15:31.309 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:15:31.309 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:15:31.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:31.363+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:15:36.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:36.364+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:15:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:41.364+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:15:46.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:46.365+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:15:51.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:51.365+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:15:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:15:56.366+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:01.366+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:06.367+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:11.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:11.368+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:16.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:16.368+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:21.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:21.369+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:26.370+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:31.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:31.370+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:36.371+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:41.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:41.371+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:46.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:46.372+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:51.372+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:16:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:16:56.373+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:01.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:01.374+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:06.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:06.374+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:11.375+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:16.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:16.376+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:21.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:21.376+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:26.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:26.377+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:31.377+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:36.378+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:41.378+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:46.379+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:51.379+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:17:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:17:56.380+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:01.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:01.380+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:06.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:06.381+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:11.381+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:16.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:16.382+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:21.383+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:26.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:26.383+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:31.384+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:36.384+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:41.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:41.385+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:46.385+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:51.386+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:18:56.387+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:01.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:01.387+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:06.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:06.388+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:11.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:11.388+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:16.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:16.389+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:21.390+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:26.390+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:31.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:31.391+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:36.392+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:41.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:41.392+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:46.393+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:51.393+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:19:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:19:56.394+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:01.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:01.395+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:06.395+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:11.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:11.396+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:16.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:16.396+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:21.397+0000 7f9db6b48a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:26.398+0000 7f9db6b48a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:20:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:26.398+0000 7f9db6b48a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:20:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:26.398+0000 7f9db6b48a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:20:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:20:26.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2024-05-26T22:20:26.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:26.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:26.444 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:20:26.500 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:26.500 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:26.536 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:20:26.549 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:20:26.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762026.442359_1045.1", 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:20:26.442359Z" 2024-05-26T22:20:26.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:26.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:26.551 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:20:26.551 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:20:26.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:26.551 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:20:26.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:20:26.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:26.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:26.552 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:20:26.610 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:26.611 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:26.644+0000 7f1dbea02a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:26.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:26.645+0000 7f1dbea02a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:26.658 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:20:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:20:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:20:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762026.434356_1042.1", 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:20:26.434356Z", 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:20:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:20:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:20:26Z" 2024-05-26T22:20:26.669 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:20:26.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716762026.442359_1045.1 target=1_1716762026.434356_1042.1 2024-05-26T22:20:31.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:31.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:31.672 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:20:31.729 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:31.729 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:31.761 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:31.761+0000 7f4c994c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:31.762 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:31.762+0000 7f4c994c3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:31.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762026.434356_1042.1", 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:20:26.434356Z", 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:20:31.779 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:20:31.780 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:20:31.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:31.781 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:20:31Z" 2024-05-26T22:20:31.781 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:20:31.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716762026.442359_1045.1 target=1_1716762026.434356_1042.1 2024-05-26T22:20:36.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:36.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:36.784 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:20:36.841 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:36.841 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:36.873 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:36.872+0000 7f98af459a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:36.873 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:36.873+0000 7f98af459a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:20:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:20:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762026.442359_1045.1", 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:20:26.442359Z", 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:20:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:20:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:20:36.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:20:36.894 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:20:36Z" 2024-05-26T22:20:36.894 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:20:36.895 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:20:36.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:20:36.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:36.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:20:36.895 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:20:36.954 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:36.954 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:36.991+0000 7fb587be6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:36.993+0000 7fb587be6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:37.008 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:20:37.009 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:20:37.010 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762026.442359_1045.1", 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:20:26.442359Z", 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:20:37.011 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:20:37.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:20:37.013 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:20:36Z" 2024-05-26T22:20:37.013 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:20:37.013 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:20:37.014 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-05-26T22:20:37.014 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T22:20:37.014 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T22:20:37.015+0000 7fd33fca4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 71322) UID: 0 2024-05-26T22:20:37.015 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T22:20:37.015+0000 7fd343f11a80 -1 shutting down 2024-05-26T22:20:43.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-05-26T22:20:43.115 INFO:tasks.rgw.c2.client.0:Stopped 2024-05-26T22:20:43.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.195 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:43.303 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:43.303 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:43.307 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 5 2024-05-26T22:20:43.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.467 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:43.567 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:43.567 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:43.573 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 10 2024-05-26T22:20:43.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.732 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:43.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:43.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:43.832 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:43.832 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:43.839 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 15 2024-05-26T22:20:43.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:43.971 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:44.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:44.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:44.073 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:44.074 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:44.082 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 20 2024-05-26T22:20:44.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.255 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:44.358 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:44.358 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:44.368 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 25 2024-05-26T22:20:44.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.555 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-75 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:44.610 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:44.610 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:44.663 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:20:44.663 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-75 2024-05-26T22:20:44.674 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 30 2024-05-26T22:20:44.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:20:44.860 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-05-26T22:20:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:44.954 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:44.954+0000 7f2a8cd91a80 5 RGW-SYNC:data: waiting on data sync status lease 2024-05-26T22:20:44.955 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:20:44.956+0000 7f2a8cd91a80 5 RGW-SYNC:data: acquired data sync status lease 2024-05-26T22:20:44.978 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-05-26T22:20:44.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-05-26T22:20:44.980 INFO:tasks.rgw.c2.client.0:Started 2024-05-26T22:20:44.980 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T22:20:45.001 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T22:20:45.001 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T22:20:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-26T22:20:45.002 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2024-05-26T22:20:46.003 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-05-26T22:20:46.028 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-26T22:20:46.028 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-05-26T22:20:46.029 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-05-26T22:20:46.029 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-05-26T22:20:46.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:20:46.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:20:46.030 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:20:46.131 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:46.131 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:46.164 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-75 2024-05-26T22:20:46.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:20:46.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:20:46.178 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:20:46.256 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:20:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:20:46.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:46.299+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:48.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-05-26T22:20:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:51.300+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:20:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:20:56.300+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:01.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:01.301+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:06.302+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:11.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:11.302+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:16.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:16.303+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:21.303+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:26.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:26.304+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:31.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:31.305+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:36.305+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:41.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:41.306+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:46.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:46.307+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:51.307+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:21:56.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:21:56.308+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:01.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:01.308+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:06.309+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:11.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:11.310+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:16.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:16.310+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:21.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:21.311+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:26.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:26.311+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:31.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:31.312+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:36.313+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:41.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:41.314+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:46.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:46.314+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:51.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:51.315+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:22:56.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:22:56.315+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:01.316+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:06.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:06.317+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:11.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:11.317+0000 7f1698f64a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:16.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:16.321+0000 7f1698f64a80 1 bucket sync caught up with source: 2024-05-26T22:23:16.321 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , 00000000002.45.6, , , , , , 00000000002.90.6, , , , , 00000000002.124.6, , , , , , , , , , , , , , , 00000000001.48.6, , , , 00000000001.46.6] 2024-05-26T22:23:16.321 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , 00000000002.45.6, , , , , , 00000000002.90.6, , , , , 00000000002.124.6, , , , , , , , , , , , , , , 00000000001.48.6, , , , 00000000001.46.6] 2024-05-26T22:23:16.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:16.321+0000 7f1698f64a80 0 bucket checkpoint complete 2024-05-26T22:23:16.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:23:16.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:23:16.342 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:23:16.408 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:23:16.408 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:23:16.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:16.448+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:21.449+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:26.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:26.449+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:31.450+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:36.451+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:41.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:41.451+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:46.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:46.452+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:51.452+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:23:56.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:23:56.453+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:01.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:01.453+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:06.454+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:11.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:11.455+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:16.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:16.455+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:21.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:21.456+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:26.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:26.456+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:31.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:31.457+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:36.458+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:41.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:41.459+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:46.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:46.459+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:51.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:51.460+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:24:56.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:24:56.460+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:01.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:01.461+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:06.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:06.461+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:11.462+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:16.463+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:21.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:21.463+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:26.464+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:31.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:31.464+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:36.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:36.465+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:41.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:41.466+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:46.466+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:51.467+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:25:56.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:25:56.467+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:01.468+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:06.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:06.468+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:11.469+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:16.469+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:21.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:21.470+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:26.471+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:31.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:31.471+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:36.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:36.472+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:41.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:41.472+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:46.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:46.473+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:51.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:51.474+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:26:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:26:56.475+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:01.475+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:06.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:06.476+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:11.477+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:16.476+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:21.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:21.477+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:26.478+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:31.478+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:36.479+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:41.479+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:46.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:46.480+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:51.480+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:27:56.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:27:56.481+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:01.481+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:06.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:06.482+0000 7f6f2fd38a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:11.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:11.483+0000 7f6f2fd38a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:28:11.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:11.483+0000 7f6f2fd38a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:28:11.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:11.483+0000 7f6f2fd38a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:28:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:28:11.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-05-26T22:28:11.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:11.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:11.528 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:28:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:28:11.634 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762491.525781_1144.1", 2024-05-26T22:28:11.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:28:11.525781Z" 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:11.636 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:28:11.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:28:11.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:11.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:11.637 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:28:11.693 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:11.694 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:11.725 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:28:11.725+0000 7f1b0bdcfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:11.726 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:28:11.726+0000 7f1b0bdcfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:11.740 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:28:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:28:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:28:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:28:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762044.683381_1122.1", 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:20:44.683381Z", 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:28:11Z" 2024-05-26T22:28:11.745 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:28:11.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716762491.525781_1144.1 target=1_1716762044.683381_1122.1 2024-05-26T22:28:16.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:16.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:16.747 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:28:16.804 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:16.804 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:16.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:28:16.837+0000 7fea10414a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:16.837 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:28:16.838+0000 7fea10414a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:16.850 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:28:16.850 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:28:16.850 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:28:16.851 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:16.852 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762491.525781_1144.1", 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:16.853 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:28:11.525781Z", 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:28:16.854 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:28:16Z" 2024-05-26T22:28:16.855 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:28:16.856 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:28:16.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:28:16.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:16.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:28:16.857 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:28:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:16.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:16.951+0000 7f699379fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:16.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:16.952+0000 7f699379fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:28:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762491.525781_1144.1", 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:28:11.525781Z", 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:28:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:28:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:28:16Z" 2024-05-26T22:28:16.971 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:28:16.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:28:16.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:16.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:16.982 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:28:17.041 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:17.041 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:17.072 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-76 2024-05-26T22:28:17.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:17.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:17.086 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:28:17.147 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:17.147 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:17.187+0000 7f7954632a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:22.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:22.186+0000 7f7954632a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:27.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:27.187+0000 7f7954632a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:32.188+0000 7f7954632a80 1 bucket sync caught up with source: 2024-05-26T22:28:32.383 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , 00000000001.69.6, , , , , ] 2024-05-26T22:28:32.383 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , 00000000001.69.6, , , , , ] 2024-05-26T22:28:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:32.188+0000 7f7954632a80 0 bucket checkpoint complete 2024-05-26T22:28:32.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:32.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:28:32.385 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:28:32.456 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:28:32.456 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:28:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:32.500+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:37.500+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:42.501+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:47.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:47.501+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:52.502+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:28:57.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:28:57.503+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:02.503+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:07.504+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:12.505+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:17.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:17.506+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:22.507+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:27.506+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:32.507+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:37.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:37.508+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:42.508+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:47.509+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:52.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:52.509+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:29:57.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:29:57.510+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:02.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:02.510+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:07.511+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:12.512+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:17.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:17.513+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:22.513+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:27.515+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:32.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:32.514+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:37.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:37.515+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:42.515+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:47.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:47.516+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:52.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:52.517+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:30:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:30:57.517+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:02.518+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:07.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:07.518+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:12.519+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:17.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:17.520+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:22.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:22.521+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:27.521+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:32.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:32.522+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:37.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:37.522+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:42.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:42.522+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:47.523+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:52.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:52.523+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:31:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:31:57.524+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:02.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:02.524+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:07.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:07.525+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:12.526+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:17.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:17.527+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:22.527+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:27.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:27.528+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:32.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:32.529+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:37.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:37.530+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:42.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:42.529+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:47.530+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:52.531+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:32:57.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:32:57.531+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:02.532+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:07.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:07.532+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:12.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:12.533+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:17.534+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:22.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:22.534+0000 7f109aeb4a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:27.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:27.535+0000 7f109aeb4a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:33:27.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:27.535+0000 7f109aeb4a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:33:27.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:27.535+0000 7f109aeb4a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:33:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:33:27.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-05-26T22:33:27.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.580 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:33:27.635 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:27.635 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:33:27.688 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:33:27.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762807.578523_1158.1", 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:33:27.578523Z" 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.690 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:33:27.690 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:33:27.690 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.690 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:33:27.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:33:27.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.691 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:33:27.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:27.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:27.777 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:33:27.776+0000 7fba038fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:27.777 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:33:27.777+0000 7fba038fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:27.790 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:33:27.790 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:33:27.790 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:33:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:33:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716762807.578523_1158.1", 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:33:27.793 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:33:27.578523Z", 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:33:27.794 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:33:27Z" 2024-05-26T22:33:27.795 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:33:27.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:33:27.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:33:27.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:27.797 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:33:27.849 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:27.850 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:27.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:27.881+0000 7f7140962a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:27.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:27.882+0000 7f7140962a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:27.896 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:33:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:33:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:33:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:33:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:33:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:33:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:33:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:33:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762807.570702_1155.1", 2024-05-26T22:33:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:33:27.570702Z", 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:33:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:33:27Z" 2024-05-26T22:33:27.906 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:33:27.907 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716762807.578523_1158.1 target=1_1716762807.570702_1155.1 2024-05-26T22:33:32.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:32.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:33:32.909 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:33:32.967 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:32.967 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:33.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:33.002+0000 7f1167a4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:33.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:33.004+0000 7f1167a4ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:33:33.019 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:33:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:33:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:33:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:33:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:33:33.022 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:33.022 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:33.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:33.022 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762807.578523_1158.1", 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:33.023 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:33:27.578523Z", 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:33:33.024 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:33:33.025 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:33:33.026 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:33:33.026 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:33:33.026 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:33.026 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:33:33.026 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:33:33Z" 2024-05-26T22:33:33.027 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:33:33.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:33:33.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.151 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:33.207 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:33.208 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:33.257 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:33.257 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:33.261 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 5 2024-05-26T22:33:33.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.427 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:33.527 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:33.527 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:33.534 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 10 2024-05-26T22:33:33.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.698 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:33.752 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:33.752 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:33.801 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:33.801 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:33.807 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 15 2024-05-26T22:33:33.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:33.971 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:34.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:34.024 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:34.075 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:34.075 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:34.082 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 20 2024-05-26T22:33:34.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:34.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:34.263 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:34.317 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:34.318 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:34.368 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:34.368 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:34.378 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 25 2024-05-26T22:33:34.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:34.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xmqxus-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:33:34.528 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xmqxus-77 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:33:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:34.634 INFO:teuthology.orchestra.run.smithi007.stdout:tenant: 2024-05-26T22:33:34.634 INFO:teuthology.orchestra.run.smithi007.stdout:bucket name: xmqxus-77 2024-05-26T22:33:34.646 INFO:teuthology.orchestra.run.smithi007.stdout:total entries: 30 2024-05-26T22:33:34.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:33:34.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:33:34.882 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:33:34.946 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:34.947 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:34.980 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-77 2024-05-26T22:33:34.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:33:34.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:33:34.995 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:33:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:33:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:33:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:35.092+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:40.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:40.093+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:45.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:45.094+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:50.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:50.095+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:33:55.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:33:55.095+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:00.096+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:05.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:05.097+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:10.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:10.097+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:15.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:15.098+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:20.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:20.098+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:25.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:25.099+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:30.099+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:35.100+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:40.101+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:45.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:45.102+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:50.102+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:34:55.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:34:55.103+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:00.104+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:05.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:05.104+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:10.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:10.105+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:15.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:15.105+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:20.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:20.106+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:25.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:25.107+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:30.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:30.108+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:35.108+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:40.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:40.109+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:45.110+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:50.111+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:35:55.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:35:55.111+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:00.112+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:05.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:05.113+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:10.113+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:15.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:15.114+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:20.114+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:25.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:25.115+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:30.116+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:35.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:35.116+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:40.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:40.117+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:45.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:45.118+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:50.118+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:36:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:36:55.119+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:00.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:00.119+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:05.120+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:10.121+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:15.121+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:20.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:20.122+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:25.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:25.123+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:30.123+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:35.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:35.124+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:40.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:40.125+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:45.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:45.126+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:50.126+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:37:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:37:55.127+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:00.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:00.128+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:05.129+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:10.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:10.129+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:15.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:15.130+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:20.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:20.131+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:25.131+0000 7f4716785a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:30.132+0000 7f4716785a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:38:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:30.132+0000 7f4716785a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:38:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:30.132+0000 7f4716785a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:38:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:38:30.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-05-26T22:38:30.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.179 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:38:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:38:30.285 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:38:30.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.285 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:38:30.285 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:38:30.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:38:30.177374Z" 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.286 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:38:30.287 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:38:30.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.287 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:38:30.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:38:30.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.288 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:38:30.341 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:30.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:30.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:38:30.372+0000 7f9c3e20da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:30.373 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:38:30.373+0000 7f9c3e20da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:30.386 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:38:30.386 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:38:30.386 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:38:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:38:30.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:38:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:38:30.390 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:38:30.391 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:38:30.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:38:30.392 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:38:30Z" 2024-05-26T22:38:30.392 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:38:30.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:38:30.393 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:38:30.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:30.393 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:38:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:30.508+0000 7f91071fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:30.509+0000 7f91071fba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.524 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716762814.656484_1235.1", 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:33:34.656484Z", 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:30.527 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:38:30.528 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:38:30.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:30.529 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:38:30Z" 2024-05-26T22:38:30.529 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:38:30.530 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763110.177374_1257.1 target=1_1716762814.656484_1235.1 2024-05-26T22:38:35.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:35.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:38:35.531 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:38:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:35.626+0000 7fbd2768aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:35.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:35.627+0000 7fbd2768aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:38:35.642 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:35.643 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:35.644 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:35.645 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:38:35.646 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:38:35.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:38:35.648 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:38:35Z" 2024-05-26T22:38:35.648 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:38:35.648 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:38:35.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:35.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:35.724 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xmqxus-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-05-26T22:38:35.787 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:35.787 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:35.820 INFO:teuthology.orchestra.run.smithi137.stdout:Sync is disabled for bucket xmqxus-78 2024-05-26T22:38:35.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:35.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:35.835 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:38:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:35.931+0000 7f1bba8e1a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:40.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:40.932+0000 7f1bba8e1a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:45.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:45.933+0000 7f1bba8e1a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:50.933+0000 7f1bba8e1a80 1 waiting to reach incremental sync.. 2024-05-26T22:38:55.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:55.935+0000 7f1bba8e1a80 1 bucket sync caught up with source: 2024-05-26T22:38:55.937 INFO:teuthology.orchestra.run.smithi007.stderr: local status: [, , , , , , , , 00000000001.44.6, 00000000002.51.6, 00000000002.167.6] 2024-05-26T22:38:55.937 INFO:teuthology.orchestra.run.smithi007.stderr: remote markers: [, , , , , , , , 00000000001.44.6, 00000000002.51.6, 00000000002.167.6] 2024-05-26T22:38:55.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:55.935+0000 7f1bba8e1a80 0 bucket checkpoint complete 2024-05-26T22:38:55.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:55.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xmqxus-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-05-26T22:38:55.979 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xmqxus-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-05-26T22:38:56.043 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:38:56.043 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:38:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:38:56.083+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:01.084+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:06.085+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:11.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:11.085+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:16.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:16.086+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:21.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:21.087+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:26.087+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:31.088+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:36.089+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:41.089+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:46.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:46.090+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:51.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:51.091+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:39:56.092+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:01.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:01.093+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:06.093+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:11.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:11.094+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:16.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:16.094+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:21.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:21.095+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:26.096+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:31.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:31.096+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:36.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:36.097+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:41.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:41.097+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:46.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:46.098+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:51.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:51.104+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:40:56.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:40:56.105+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:01.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:01.105+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:06.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:06.106+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:11.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:11.106+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:16.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:16.107+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:21.107+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:26.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:26.108+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:31.108+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:36.109+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:41.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:41.110+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:46.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:46.110+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:51.111+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:41:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:41:56.111+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:01.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:01.113+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:06.114+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:11.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:11.114+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:16.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:16.115+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:21.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:21.116+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:26.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:26.116+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:31.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:31.117+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:36.117+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:41.118+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:46.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:46.118+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:51.118+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:42:56.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:42:56.119+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:01.120+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:06.121+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:11.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:11.121+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:16.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:16.122+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:21.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:21.122+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:26.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:26.123+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:31.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:31.123+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:36.124+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:41.124+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:46.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:46.125+0000 7f49658d3a80 1 waiting to reach incremental sync.. 2024-05-26T22:43:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:51.125+0000 7f49658d3a80 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-05-26T22:43:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:51.125+0000 7f49658d3a80 0 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:43:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:51.125+0000 7f49658d3a80 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-05-26T22:43:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-26T22:43:51.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-05-26T22:43:51.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.145 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.246 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:38:30.177374Z" 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.247 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:43:51.248 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:43:51.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.248 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:51.304 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.305 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.338 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:51.337+0000 7fe6c1f33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:51.338 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:51.338+0000 7fe6c1f33a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:51.351 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:43:51.351 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:43:51.351 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:43:51.351 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:43:51.351 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:51.352 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:43:51.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:43:51.354 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:51.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:43:51Z" 2024-05-26T22:43:51.356 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:43:51.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:43:51.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:43:51.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:51.358 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:51.442+0000 7f31ec8e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:51.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:43:51.443+0000 7f31ec8e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:51.457 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:43:51.457 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:43:51.457 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:43:51.457 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:43:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:43:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:43:51.460 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:43:51Z" 2024-05-26T22:43:51.462 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:43:51.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:43:51.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.464 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.510 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.510 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:43:51.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.556 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.607 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.608 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.654 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:43:51.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.655 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.751 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:51.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.752 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.800 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-05-26T22:43:51.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.846 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.893 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:51.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:51.939 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:52.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.041 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:52.098 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:52.145 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:52.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.146 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:52.245 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:43:52.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.246 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:52.294 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:52.295 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:52.340 INFO:teuthology.orchestra.run.smithi007.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-05-26T22:43:52.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.341 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 7, 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:43:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:43:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:43:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:43:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:43:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:43:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:43:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:43:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:43:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:43:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:43:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:43:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:43:52.463 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:43:52.464 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-79 2024-05-26T22:43:52.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:52.880 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.104 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:43:53.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.105 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.281 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:53.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.282 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.349 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.350 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.414 INFO:teuthology.orchestra.run.smithi007.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-05-26T22:43:53.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.414 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.511 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:53.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.511 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.618 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:53.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.619 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.677 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.677 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.728 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:43:53.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.728 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.836 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T22:43:53.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:43:53.837 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:43:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:53.936 INFO:teuthology.orchestra.run.smithi007.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-05-26T22:43:53.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:53.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:53.936 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:53.985 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:53.985 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:54.017 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:43:54.030 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:43:54.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:54.030 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:54.030 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:43:54.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763433.823205_1294.1", 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:43:53.823205Z" 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:54.031 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:43:54.032 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:43:54.032 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:43:54.033 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:43:54.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:54.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:54.033 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:54.085 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:54.085 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:54.115 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:54.116+0000 7f5a06a57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:54.116 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:54.116+0000 7f5a06a57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:54.129 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:43:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:43:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:43:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:43:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:43:54Z" 2024-05-26T22:43:54.134 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:43:54.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:43:59.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:59.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:43:59.137 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:43:59.189 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:43:59.189 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:43:59.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:59.220+0000 7fb4bccbba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:59.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:43:59.221+0000 7fb4bccbba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:43:59.233 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:59.234 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:59.235 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:59.236 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:43:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:43:59.238 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:43:59Z" 2024-05-26T22:43:59.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:43:59.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:04.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:04.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:04.241 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:04.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:04.299 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:04.330 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:04.331+0000 7f3a82848a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:04.331 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:04.331+0000 7f3a82848a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:04.345 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:04.345 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:04.345 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:04.345 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:04.345 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:04.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:04.347 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:04.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:04.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:04Z" 2024-05-26T22:44:04.350 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:04.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:09.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:09.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:09.352 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:09.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:09.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:09.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:09.440+0000 7f5533578a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:09.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:09.441+0000 7f5533578a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:09Z" 2024-05-26T22:44:09.460 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:09.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:14.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:14.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:14.462 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:14.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:14.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:14.552 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:14.553+0000 7f7a0f4aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:14.552 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:14.554+0000 7f7a0f4aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:14.566 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:14Z" 2024-05-26T22:44:14.571 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:14.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:19.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:19.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:19.574 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:19.631 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:19.632 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:19.663 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:19.665+0000 7f9586f6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:19.664 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:19.666+0000 7f9586f6ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:19.678 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:19.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:19.682 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:19.683 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:19.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:19.683 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:19Z" 2024-05-26T22:44:19.683 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:19.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:24.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:24.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:24.686 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:24.743 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:24.743 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:24.777 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:24.778+0000 7fb90f510a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:24.778 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:24.779+0000 7fb90f510a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:24.791 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:24.791 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:24.792 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:24.793 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:24.794 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:24.795 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:24.796 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:24.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:24.797 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:24Z" 2024-05-26T22:44:24.797 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:24.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:29.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:29.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:29.799 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:29.856 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:29.856 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:29.888 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:29.888+0000 7f172df1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:29.889 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:29.889+0000 7f172df1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.903 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:29.904 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:29.905 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:29.906 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:29.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:29.908 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:29Z" 2024-05-26T22:44:29.908 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:29.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:34.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:34.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:34.910 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:34.966 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:34.966 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:34.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:34.998+0000 7fc4d235da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:34.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:34.999+0000 7fc4d235da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:35.012 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:35.012 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:35.013 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:35.014 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:35.015 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:35.016 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:35Z" 2024-05-26T22:44:35.017 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:35.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:40.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:40.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:40.019 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:40.075 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:40.075 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:40.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:40.110+0000 7fe255a3ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:40.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:40.111+0000 7fe255a3ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:40.124 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:40.124 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:40.124 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:40.125 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:40.126 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:40.128 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:40Z" 2024-05-26T22:44:40.129 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:40.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:45.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:45.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:45.131 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:45.187 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:45.187 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:45.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:45.220+0000 7fb9bc847a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:45.221+0000 7fb9bc847a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:45.234 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:45.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:45Z" 2024-05-26T22:44:45.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:45.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:50.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:50.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:50.242 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:50.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:50.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:50.330 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:50.330+0000 7fdabe5aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:50.331 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:50.331+0000 7fdabe5aea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:50.344 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:50.345 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:50.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:50.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:50.348 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:50Z" 2024-05-26T22:44:50.349 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:50.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:44:55.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:55.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:44:55.352 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:44:55.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:44:55.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:44:55.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:55.440+0000 7f8627906a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:55.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:44:55.441+0000 7f8627906a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:44:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:55.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:55.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:44:55.459 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:44:55Z" 2024-05-26T22:44:55.460 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:44:55.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:00.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:00.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:00.462 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:00.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:00.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:00.552+0000 7f5805143a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:00.553+0000 7f5805143a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:00.567 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:00.568 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:00.569 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:00.570 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:00.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:00Z" 2024-05-26T22:45:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:00.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:05.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:05.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:05.574 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:05.631 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:05.631 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:05.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:05.667+0000 7f4a421f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:05.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:05.669+0000 7f4a421f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:05Z" 2024-05-26T22:45:05.688 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:05.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:10.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:10.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:10.690 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:10.748 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:10.748 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:10.781 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:10.782+0000 7f8202654a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:10.781 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:10.783+0000 7f8202654a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:10.795 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:10.795 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:10.795 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:10.795 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:10.796 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:10.797 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:10.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:10Z" 2024-05-26T22:45:10.800 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:10.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:15.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:15.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:15.802 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:15.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:15.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:15.893+0000 7f1290ecba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:15.894+0000 7f1290ecba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:15.906 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:15Z" 2024-05-26T22:45:15.911 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:15.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:20.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:20.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:20.914 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:20.970 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:20.970 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:21.004+0000 7f34bed3ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:21.005+0000 7f34bed3ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:21.017 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:21.017 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:21.017 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:21.017 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:21.017 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:21.018 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:21.020 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:21.021 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:21Z" 2024-05-26T22:45:21.022 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:21.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:26.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:26.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:26.024 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:26.083 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:26.083 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:26.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:26.127+0000 7f5e4fc52a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:26.128 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:26.128+0000 7f5e4fc52a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:26.141 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:26Z" 2024-05-26T22:45:26.146 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:26.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:31.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:31.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:31.149 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:31.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:31.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:31.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:31.238+0000 7f51b7ee5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:31.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:31.239+0000 7f51b7ee5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:31.252 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:31.252 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:31.252 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:31.253 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:31.254 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:31.255 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:31.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:31Z" 2024-05-26T22:45:31.257 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:31.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:36.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:36.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:36.259 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:36.316 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:36.316 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:36.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:36.349+0000 7f4c05c3ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:36.349 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:36.350+0000 7f4c05c3ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:36.363 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:36.364 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:36.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:36.366 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:36.369 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:36.369 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:36.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:36.369 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:36Z" 2024-05-26T22:45:36.369 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:36.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:41.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:41.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:41.371 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:41.427 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:41.427 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:41.458 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:41.458+0000 7fd97ba57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:41.459 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:41.459+0000 7fd97ba57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:41.473 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:41.473 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:41.473 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:41.473 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:41.473 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:41.474 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:41.474 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:41.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.474 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:41.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:41.475 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:41.476 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:41.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:41.478 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:41Z" 2024-05-26T22:45:41.479 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:41.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:46.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:46.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:46.481 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:46.538 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:46.538 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:46.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:46.570+0000 7f1618cfaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:46.571 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:46.571+0000 7f1618cfaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:46Z" 2024-05-26T22:45:46.590 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:46.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:51.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:51.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:51.592 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:51.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:51.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:51.681 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:51.682+0000 7f95850eca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:51.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:51.683+0000 7f95850eca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:51.696 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:51.696 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:51.696 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:51.696 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:51.696 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:51.697 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:51.698 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:51.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:51.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:51Z" 2024-05-26T22:45:51.701 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:51.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:45:56.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:56.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:45:56.703 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:45:56.761 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:45:56.761 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:45:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:56.793+0000 7efd49078a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:45:56.794+0000 7efd49078a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:45:56Z" 2024-05-26T22:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:45:56.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:46:01.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:01.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:01.817 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:01.875 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:01.875 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:01.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:01.907+0000 7fcf67a9ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:01.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:01.908+0000 7fcf67a9ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:01.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763110.177374_1257.1", 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:30.177374Z", 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:01.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:01.926 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:01Z" 2024-05-26T22:46:01.927 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:01.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763110.177374_1257.1 2024-05-26T22:46:06.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:06.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:06.929 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:06.991 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:06.991 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:07.022 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:07.023+0000 7f78a6bfba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:07.022 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:07.024+0000 7f78a6bfba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:07.035 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763432.842794_1270.1", 2024-05-26T22:46:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:43:52.842794Z", 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:07Z" 2024-05-26T22:46:07.040 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:07.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763433.823205_1294.1 target=1_1716763432.842794_1270.1 2024-05-26T22:46:12.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.043 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:12.098 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.098 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.130 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:12.131+0000 7ff6fd1b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:12.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:12.132+0000 7ff6fd1b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:12.144 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:12.144 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:12.145 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:12.146 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763433.823205_1294.1", 2024-05-26T22:46:12.147 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:43:53.823205Z", 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:12.148 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:12Z" 2024-05-26T22:46:12.149 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:12.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:46:12.151 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:46:12.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.151 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:12.246+0000 7fb0bec2aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:12.247+0000 7fb0bec2aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:12.263 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:46:12.263 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:12.264 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763433.823205_1294.1", 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:12.266 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:43:53.823205Z", 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:12.267 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:46:12Z" 2024-05-26T22:46:12.268 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:46:12.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:46:12.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.270 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.373 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:12.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.373 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:12.430 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.431 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:12.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.484 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:12.542 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.543 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.592 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:12.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.593 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.695 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:46:12.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xmqxus-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:12.695 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xmqxus-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:12.744 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.744 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:46:12.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.791 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:12.840 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.840 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:12.886 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:46:12.912 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:46:12.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.912 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:46:12.681693Z" 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:46:12.913 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:12.914 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:46:12.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:46:12.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:12.915 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:12.979 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:12.980 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:13.010 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:13.011+0000 7f917d0dca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:13.011 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:13.012+0000 7f917d0dca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:13.024 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:13.024 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:13.024 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:13.025 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:13.026 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763433.823205_1294.1", 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:43:53.823205Z", 2024-05-26T22:46:13.027 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:13.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:13Z" 2024-05-26T22:46:13.029 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:13.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763572.681693_1322.1 target=1_1716763433.823205_1294.1 2024-05-26T22:46:18.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.032 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:18.084 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.084 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:18.114+0000 7f82dfcf0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:18.114 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:18.115+0000 7f82dfcf0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:18.127 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:18.127 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:18.127 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:18.127 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:18.127 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:18.128 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:18.129 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:18.130 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:18.131 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:18.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:18.133 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:18Z" 2024-05-26T22:46:18.133 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:18.134 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:46:18.134 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:46:18.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.134 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:18.230+0000 7fe634764a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:18.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:18.231+0000 7fe634764a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:18.250 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:46:18.250 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:18.251 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:18.252 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.253 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:18.254 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:46:18.255 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:46:18.256 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:46:18.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.256 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:46:18Z" 2024-05-26T22:46:18.256 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:46:18.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:46:18.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.257 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.310 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.310 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.357 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:18.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.357 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.462 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:18.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.462 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.515 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.516 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-05-26T22:46:18.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.565 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.615 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.615 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.666 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:46:18.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.667 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.770 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:46:18.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:18.771 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:18.819 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.819 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:18.871 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:46:18.871 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:18.871 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 8, 2024-05-26T22:46:18.871 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:46:18.871 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.872 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:18.873 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:18.874 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:18.875 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:18.876 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:18.877 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:46:18.878 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:46:18.879 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:18.880 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:18.881 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:46:18.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:18.885 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:18.885 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:46:18.886 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:46:18.887 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:18.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:18.889 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:18.890 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:18.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:18.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:18.891 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:46:18.891 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:46:18.892 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:18.892 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:46:18.893 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-05-26T22:46:18.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:18.894 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:46:19.042 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:46:19.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.044 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:19.044 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:46:19.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:46:12.681693Z" 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.045 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:46:19.047 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:46:19.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:19.047 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:19.101 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:19.101 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:19.167 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:19.168+0000 7fa57a8a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:19.168 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:19.169+0000 7fa57a8a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:19.197 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:19.198 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:19.199 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:19.200 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:19.201 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:19.203 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:19.203 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:19.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:19.203 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:19Z" 2024-05-26T22:46:19.203 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:19.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:46:19.204 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:46:19.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:19.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:19.204 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:19.309 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:19.309 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:19.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:19.494+0000 7fb3475e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:19.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:46:19.496+0000 7fb3475e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:19.535 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:19.536 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:19.537 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.538 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:46:19.538 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:19.538 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.539 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:46:19.540 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:46:19.541 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:46:19.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:19.542 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:46:19Z" 2024-05-26T22:46:19.542 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:46:19.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:46:19.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.543 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:19.611 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:19.612 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:19.749 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:46:19.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.750 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:19.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:19.821 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:19.902 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:46:19.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:19.903 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.018 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:46:20.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.018 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.123 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:46:20.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.123 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.222 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:46:20.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.222 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.321 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:46:20.321 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:20.321 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 9, 2024-05-26T22:46:20.321 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:46:20.321 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.322 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:20.323 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:46:20.326 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:46:20.327 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:46:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:46:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:46:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:46:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:46:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:46:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:46:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:46:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:46:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:46:20.343 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:46:20.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:46:20.344 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:46:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.446 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:46:20.447 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-80 2024-05-26T22:46:20.570 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-81 2024-05-26T22:46:20.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:20.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:20.749 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.892 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:46:20.679697Z" 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:46:20.893 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:46:20.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:46:20.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:20.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:20.895 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:20.950 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:20.950 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:20.981 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:20.981+0000 7f427bd57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:20.982 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:20.982+0000 7f427bd57a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:20.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:20.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:20.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:20.999 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:21.000 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:21.001 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:21.001 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:21.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:21.001 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:20Z" 2024-05-26T22:46:21.001 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:21.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:26.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:26.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:26.003 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:26.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:26.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:26.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:26.096+0000 7fe8b3d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:26.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:26.097+0000 7fe8b3d63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:26.111 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:26Z" 2024-05-26T22:46:26.117 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:26.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:31.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:31.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:31.120 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:31.178 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:31.179 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:31.211 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:31.211+0000 7feee8740a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:31.211 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:31.212+0000 7feee8740a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:31.225 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:31.225 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:31.226 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:31.227 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:31.228 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:31.229 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:31Z" 2024-05-26T22:46:31.230 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:31.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:36.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:36.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:36.233 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:36.289 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:36.289 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:36.322 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:36.322+0000 7fd17b6eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:36.322 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:36.323+0000 7fd17b6eaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:36.339 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:36.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:36Z" 2024-05-26T22:46:36.342 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:36.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:41.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:41.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:41.345 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:41.402 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:41.402 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:41.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:41.435+0000 7fdcb09f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:41.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:41.436+0000 7fdcb09f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:41.449 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:41.450 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:41Z" 2024-05-26T22:46:41.454 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:41.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:46.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:46.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:46.457 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:46.514 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:46.514 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:46.546 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:46.546+0000 7f892c426a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:46.547 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:46.547+0000 7f892c426a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:46.562 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:46.563 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:46.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:46.565 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:46Z" 2024-05-26T22:46:46.566 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:46.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:51.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:51.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:51.568 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:51.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:51.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:51.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:51.661+0000 7f139f454a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:51.662 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:51.662+0000 7f139f454a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:51.676 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:51.676 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:51.676 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:51.676 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:51.676 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:51.677 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:51.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:51.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:51.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:51Z" 2024-05-26T22:46:51.681 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:51.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:46:56.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:56.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:46:56.683 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:46:56.740 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:46:56.740 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:46:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:56.774+0000 7fd7771fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:56.775 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:46:56.775+0000 7fd7771fda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:46:56.791 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:46:56.791 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:46:56.791 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:46:56.791 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:46:56.791 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:46:56.792 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:56.793 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:56.794 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:56.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:56.796 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:56.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:46:56.798 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:56.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:46:56Z" 2024-05-26T22:46:56.800 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:46:56.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:01.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:01.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:01.803 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:01.860 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:01.860 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:01.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:01.894+0000 7f3eceb82a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:01.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:01.895+0000 7f3eceb82a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:01Z" 2024-05-26T22:47:01.917 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:01.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:06.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:06.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:06.921 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:06.978 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:06.978 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:07.012 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:07.012+0000 7fbe6ebb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:07.013 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:07.013+0000 7fbe6ebb2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:07.026 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:07Z" 2024-05-26T22:47:07.031 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:07.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:12.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:12.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:12.034 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:12.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:12.091 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:12.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:12.124+0000 7f46de7efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:12.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:12.125+0000 7f46de7efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.138 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:12.140 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:12.141 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:12.142 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:12.143 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:12.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:12.143 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:12Z" 2024-05-26T22:47:12.143 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:12.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:17.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:17.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:17.145 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:17.202 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:17.203 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:17.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:17.236+0000 7fb7897e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:17.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:17.237+0000 7fb7897e7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:17.255 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:17.255 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:17.255 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:17.255 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:17.255 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:17.256 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:17.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:17.259 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:17.260 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:17Z" 2024-05-26T22:47:17.261 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:17.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:22.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:22.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:22.262 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:22.320 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:22.320 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:22.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:22.352+0000 7fb7e0faba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:22.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:22.353+0000 7fb7e0faba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:22.367 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:22.368 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:22.369 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:22.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:22.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:22Z" 2024-05-26T22:47:22.372 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:22.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:27.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:27.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:27.374 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:27.430 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:27.431 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:27.464 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:27.464+0000 7f8fa1802a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:27.465 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:27.465+0000 7f8fa1802a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:27.479 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:27.479 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:27.480 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:27.481 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:27.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:27.483 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:27.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:27Z" 2024-05-26T22:47:27.485 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:27.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:32.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:32.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:32.488 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:32.545 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:32.546 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:32.578 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:32.578+0000 7fa01215aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:32.579 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:32.579+0000 7fa01215aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:32.593 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:32.594 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:32.595 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:32.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:32.597 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:32.598 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:32Z" 2024-05-26T22:47:32.599 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:32.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:37.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:37.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:37.601 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:37.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:37.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:37.694 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:37.693+0000 7efe85a12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:37.694 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:37.694+0000 7efe85a12a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:37.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.710 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:37.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:37.710 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:37.710 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:37.710 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:37.713 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:37.714 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:37.715 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:37.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:37.717 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:37.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:37.718 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:37Z" 2024-05-26T22:47:37.718 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:37.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:42.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:42.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:42.721 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:42.779 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:42.779 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:42.812 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:42.812+0000 7f1f5b114a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:42.812 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:42.813+0000 7f1f5b114a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:42.827 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:42.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:42.830 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:42.831 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:42.832 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:42Z" 2024-05-26T22:47:42.832 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:42.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:47.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:47.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:47.835 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:47.891 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:47.891 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:47.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:47.923+0000 7f0053969a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:47.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:47.924+0000 7f0053969a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.939 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:47.942 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:47.943 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:47Z" 2024-05-26T22:47:47.944 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:47.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:52.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:52.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:52.947 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:53.003 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:53.004 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:53.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:53.036+0000 7fa753261a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:53.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:53.036+0000 7fa753261a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:53.051 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:53.053 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:53.054 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:53Z" 2024-05-26T22:47:53.055 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:53.056 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:47:58.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:58.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:47:58.058 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:47:58.115 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:47:58.115 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:47:58.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:58.149+0000 7ff7243c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:58.149 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:47:58.150+0000 7ff7243c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:47:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:47:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:47:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:47:58Z" 2024-05-26T22:47:58.168 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:47:58.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:03.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:03.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:03.170 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:03.226 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:03.226 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:03.258 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:03.259+0000 7ff68c541a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:03.259 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:03.260+0000 7ff68c541a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:03.273 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:03.273 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:03.273 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:03.273 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:03.273 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:03.274 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:03Z" 2024-05-26T22:48:03.278 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:03.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:08.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:08.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:08.281 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:08.338 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:08.338 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:08.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:08.372+0000 7fdede2dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:08.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:08.373+0000 7fdede2dda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:08.385 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:08.385 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:08.385 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:08.387 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:08.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:08.389 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:08Z" 2024-05-26T22:48:08.390 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:08.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:13.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:13.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:13.393 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:13.450 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:13.451 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:13.482 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:13.482+0000 7fd52633aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:13.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:13.483+0000 7fd52633aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:13.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:13.499 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:13.500 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:13.501 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:13.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:13Z" 2024-05-26T22:48:13.503 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:13.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:48:13.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:48:13.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:13.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:13.505 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:13.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:13.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:13.600+0000 7f3e84ff7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:13.601+0000 7f3e84ff7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:13.616 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:13.617 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:13.618 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:13.619 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:13.620 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:13.621 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:13Z" 2024-05-26T22:48:13.622 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:13.623 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:18.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:18.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:18.624 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:18.685 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:18.686 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:18.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:18.720+0000 7f26f556aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:18.721+0000 7f26f556aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:18.736 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:18.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:18.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:18.741 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:18.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:18.741 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:18Z" 2024-05-26T22:48:18.741 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:18.742 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:23.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:23.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:23.743 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:23.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:23.837+0000 7f9f66a8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:23.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:23.838+0000 7f9f66a8da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:23.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:23.856 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:23.857 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:23Z" 2024-05-26T22:48:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:23.859 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:28.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:28.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:28.861 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:28.956+0000 7f397d9c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:28.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:28.957+0000 7f397d9c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:28.976 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:28.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:28.978 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763572.681693_1322.1", 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:12.681693Z", 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:28.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:28.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:28Z" 2024-05-26T22:48:28.981 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:28.982 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763580.679697_1348.1 target=1_1716763572.681693_1322.1 2024-05-26T22:48:33.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:33.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:33.984 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:34.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:34.079+0000 7f2b14f0fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:34.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:34.080+0000 7f2b14f0fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:34.095 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:34.096 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:34.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.099 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:34Z" 2024-05-26T22:48:34.100 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:34.101 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:48:34.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:34.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:34.101 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014223", 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:33:38.450498Z" 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000024246", 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:46:20.569961Z" 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000019338", 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:48:21.804791Z" 2024-05-26T22:48:34.206 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:34.207 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:48:34.207 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:48:34.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:34.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:34.208 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:34.266 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:34.267 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:34.317 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014223", 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:33:38.450498Z" 2024-05-26T22:48:34.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024246", 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.569961Z" 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:34.320 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019196", 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.459497Z" 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:34.321 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:48:34.322 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:48:34.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:34.322 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:34Z" 2024-05-26T22:48:34.322 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:34.323 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=00000000000000000000:00000000000000019338 target=00000000000000000000:00000000000000019196 2024-05-26T22:48:39.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:39.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:39.324 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:39.383 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:39.383 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:39.433 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014223", 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:33:38.450498Z" 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024246", 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.569961Z" 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019196", 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.459497Z" 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:39Z" 2024-05-26T22:48:39.438 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:39.439 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=00000000000000000000:00000000000000019338 target=00000000000000000000:00000000000000019196 2024-05-26T22:48:44.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.441 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:44.499 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:44.500 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:44.550 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:44.550 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:44.550 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:44.550 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:44.550 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014223", 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:33:38.450498Z" 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024246", 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.552 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.569961Z" 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019338", 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:21.804791Z" 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.554 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:44Z" 2024-05-26T22:48:44.555 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:44.556 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:48:44.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:44.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:44.573 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:44.629 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:44.629 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:44.681 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:48:44.682 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2024-05-26T22:48:44.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.683 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:44.765 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:46:20.679697Z" 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.780 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:48:44.781 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:48:44.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.782 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:44.834 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:44.834 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:44.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:44.865+0000 7f76f4ae0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:44.865 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:44.866+0000 7f76f4ae0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:44.878 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:44.878 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:44.879 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.881 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:44.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:44Z" 2024-05-26T22:48:44.883 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:44.884 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:48:44.885 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:48:44.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:44.885 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:44.935 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:44.936 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:44.968+0000 7fba50069a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:48:44.969+0000 7fba50069a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:44.983 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:48:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:48:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:48:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:48:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:48:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:48:44Z" 2024-05-26T22:48:44.989 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:44.990 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:48:44.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:44.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:44.990 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.083 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:48:45.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.083 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.133 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.134 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.181 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:48:45.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.181 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.284 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:48:45.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.285 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.338 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.338 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.387 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:48:45.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.388 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.487 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:48:45.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.488 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.537 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.537 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.588 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 10, 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:48:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:48:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:48:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:48:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:48:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow", 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:48:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:48:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:48:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:48:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:48:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:48:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:48:45.609 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:48:45.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:48:45.611 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:48:45.658 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:45.658 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:45.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:48:45.705 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-82 2024-05-26T22:48:46.138 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-83 2024-05-26T22:48:46.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:46.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:46.267 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:46.339 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:46.339 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:48:46.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:48:46.190820Z" 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:48:46.424 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:48:46.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:48:46.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:46.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:46.426 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:46.475 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:46.475 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:46.505 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:46.505+0000 7f1bf1eeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:46.505 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:46.506+0000 7f1bf1eeca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:46.522 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:46.522 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:46.522 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:46.523 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:46.524 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:46.525 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:46.526 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:46Z" 2024-05-26T22:48:46.527 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:46.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:48:51.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:51.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:51.529 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:51.619 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:51.619+0000 7f89ce36ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:51.620 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:51.620+0000 7f89ce36ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:51.638 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:51.638 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:51.639 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:51.640 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:51.641 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:51.642 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:51Z" 2024-05-26T22:48:51.643 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:51.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:48:56.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:56.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:48:56.644 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:48:56.702 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:48:56.702 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:48:56.734 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:56.736+0000 7fc787e22a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:56.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:48:56.737+0000 7fc787e22a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:48:56.752 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:56.753 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:48:56.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:56.753 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:48:56.753 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:56.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:56.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:56.754 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:48:56.754 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:56.754 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:56.755 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:48:56.755 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:56.755 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:56.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:56.755 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:48:56.756 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:56.756 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:56.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:56.756 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:56.757 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:48:56.759 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:48:56.759 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:48:56.759 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:48:56.760 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:48:56.760 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:48:56.760 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:48:56.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:48:56.760 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.761 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:48:56.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:48:56.762 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:48:56.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:48:56.762 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:48:56Z" 2024-05-26T22:48:56.762 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:48:56.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:01.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:01.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:01.766 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:01.823 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:01.823 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:01.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:01.856+0000 7fc7279d4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:01.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:01.857+0000 7fc7279d4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:01.870 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:01.870 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:01.870 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:01.872 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:01.873 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:01Z" 2024-05-26T22:49:01.875 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:01.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:06.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:06.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:06.878 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:06.936 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:06.936 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:06.968 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:06.969+0000 7f99354c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:06.969 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:06.970+0000 7f99354c5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:06.983 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:06Z" 2024-05-26T22:49:06.989 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:06.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:11.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:11.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:11.991 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:12.048 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:12.048 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:12.081 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:12.081+0000 7f53233b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:12.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:12.082+0000 7f53233b8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:12.099 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:12.100 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:12.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:12Z" 2024-05-26T22:49:12.102 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:12.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:17.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:17.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:17.104 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:17.161 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:17.161 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:17.194 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:17.194+0000 7f358805ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:17.195 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:17.195+0000 7f358805ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:17.209 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:17.212 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:17.213 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:17Z" 2024-05-26T22:49:17.214 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:17.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:22.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:22.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:22.217 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:22.273 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:22.273 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:22.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:22.307+0000 7fbfccf09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:22.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:22.308+0000 7fbfccf09a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:22.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:22.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:22Z" 2024-05-26T22:49:22.327 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:22.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:27.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:27.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:27.330 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:27.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:27.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:27.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:27.420+0000 7fd85db3ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:27.420 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:27.421+0000 7fd85db3ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:27.437 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:27.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:27Z" 2024-05-26T22:49:27.440 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:27.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:32.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:32.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:32.442 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:32.499 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:32.499 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:32.534 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:32.534+0000 7f58f07f3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:32.534 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:32.535+0000 7f58f07f3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:32.550 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:32.551 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:32.553 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:32.553 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:32.553 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:32.553 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:32.553 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:32.554 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:32.554 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:32.554 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:32.554 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:32.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:32.555 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:32.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:32.558 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:32Z" 2024-05-26T22:49:32.558 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:32.559 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:37.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:37.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:37.561 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:37.653 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:37.653+0000 7f61ca9dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:37.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:37.654+0000 7f61ca9dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:37Z" 2024-05-26T22:49:37.673 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:37.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:42.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:42.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:42.676 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:42.734 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:42.734 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:42.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:42.769+0000 7f977cc59a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:42.770 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:42.770+0000 7f977cc59a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:42.784 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:42.789 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:42Z" 2024-05-26T22:49:42.790 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:42.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:47.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:47.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:47.792 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:47.849 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:47.849 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:47.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:47.882+0000 7f3e1773ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:47.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:47.883+0000 7f3e1773ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:47.897 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:47.897 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:47.897 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:47.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:47Z" 2024-05-26T22:49:47.905 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:47.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:52.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:52.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:52.908 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:52.965 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:52.965 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:52.998 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:52.999+0000 7f7161c68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:52.999 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:53.000+0000 7f7161c68a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:53.016 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:53Z" 2024-05-26T22:49:53.017 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:53.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:49:58.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:58.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:49:58.020 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:49:58.077 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:49:58.077 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:49:58.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:58.111+0000 7f7a1b7daa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:58.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:49:58.112+0000 7f7a1b7daa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:49:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:49:58Z" 2024-05-26T22:49:58.130 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:49:58.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:03.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:03.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:03.132 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:03.189 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:03.190 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:03.223 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:03.224+0000 7f85a9422a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:03.223 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:03.225+0000 7f85a9422a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:03.243 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:03.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:03.243 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:03Z" 2024-05-26T22:50:03.243 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:03.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:08.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:08.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:08.245 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:08.302 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:08.303 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:08.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:08.335+0000 7f4582bf2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:08.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:08.336+0000 7f4582bf2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:50:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:08Z" 2024-05-26T22:50:08.355 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:08.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:13.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:13.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:13.358 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:13.415 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:13.415 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:13.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:13.448+0000 7f1e0623aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:13.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:13.449+0000 7f1e0623aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:13.463 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:13Z" 2024-05-26T22:50:13.468 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:13.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:18.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:18.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:18.470 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:18.528 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:18.528 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:18.561 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:18.561+0000 7f0421a54a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:18.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:18.562+0000 7f0421a54a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:18.576 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:50:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:18.581 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:18.582 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:18.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:18.582 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:18Z" 2024-05-26T22:50:18.582 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:18.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:23.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:23.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:23.584 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:23.641 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:23.642 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:23.677+0000 7f51fcc47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:23.678+0000 7f51fcc47a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:23.695 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763580.679697_1348.1", 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.679697Z", 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:23.696 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:23Z" 2024-05-26T22:50:23.698 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:23.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763580.679697_1348.1 2024-05-26T22:50:28.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:28.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:28.701 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:28.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:28.791+0000 7efdb5c88a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:28.792 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:28.793+0000 7efdb5c88a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:28.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:28.808 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:28.809 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763725.993746_1362.1", 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:45.993746Z", 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:28Z" 2024-05-26T22:50:28.812 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:28.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763725.993746_1362.1 2024-05-26T22:50:33.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:33.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:33.814 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:33.872 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:33.872 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:33.904+0000 7fed71ac6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:33.905+0000 7fed71ac6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:33.919 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:33.919 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:33.919 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:33.919 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:33.920 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:33.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763725.993746_1362.1", 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:45.993746Z", 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:33.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:33Z" 2024-05-26T22:50:33.924 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:33.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763725.993746_1362.1 2024-05-26T22:50:38.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:38.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:38.927 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:38.984 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:38.984 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:39.017 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:39.017+0000 7f1e1c677a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:39.018 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:50:39.018+0000 7f1e1c677a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:39.032 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:50:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:50:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:39Z" 2024-05-26T22:50:39.043 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:39.044 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:50:39.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:50:39.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:39.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:39.044 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:50:39.140+0000 7f5e928f1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:50:39.141+0000 7f5e928f1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:39.156 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:50:39.156 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:50:39.156 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.157 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:39.158 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:39.160 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:39.161 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763725.993746_1362.1", 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:39.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:45.993746Z", 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:39.163 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:39.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:50:39.165 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:50:39.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:39.166 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:50:39Z" 2024-05-26T22:50:39.166 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:50:39.167 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716763726.190820_1376.1 target=1_1716763725.993746_1362.1 2024-05-26T22:50:44.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.169 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:44.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:50:44.264+0000 7f84e0ecfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:44.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:50:44.265+0000 7f84e0ecfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:50:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:50:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:50:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:50:44Z" 2024-05-26T22:50:44.285 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:50:44.286 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:50:44.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.286 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:44.338 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:44.338 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:44.392 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000024528", 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:48:46.106066Z" 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000019620", 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.100962Z" 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:44.394 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:50:44.395 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:50:44.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:44.396 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:44.455 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:44.455 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:44.506 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:44.506 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:44.506 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:44.506 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014223", 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:44.507 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:33:38.450498Z" 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024246", 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:46:20.569961Z" 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:44.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:44.509 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019338", 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:21.804791Z" 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:50:44.511 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:50:44.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:44.511 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:44Z" 2024-05-26T22:50:44.511 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:44.512 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014364 target=00000000000000000000:00000000000000014223, shard=1 master=00000000000000000000:00000000000000024528 target=00000000000000000000:00000000000000024246, shard=3 master=00000000000000000000:00000000000000019620 target=00000000000000000000:00000000000000019338 2024-05-26T22:50:49.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.513 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:49.572 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:49.573 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.623 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:49.624 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024528", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.106066Z" 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:49.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019620", 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:50:49.626 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.100962Z" 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:50:49Z" 2024-05-26T22:50:49.627 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:50:49.628 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:50:49.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.629 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:49.678 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:49.679 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout:[ 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000011651", 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T22:50:47.180399Z" 2024-05-26T22:50:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000007584", 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_update": "2024-05-26T22:50:47.145967Z" 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:50:49.730 INFO:teuthology.orchestra.run.smithi137.stdout:] 2024-05-26T22:50:49.731 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-05-26T22:50:49.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:49.732 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:49.846 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:50:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:50:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:50:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:50:49.846 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:49.847 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011369", 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.222020Z" 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:50:49.849 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:49.850 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007443", 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.267505Z" 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:49.851 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:50:49Z" 2024-05-26T22:50:49.852 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:50:49.852 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000011651 target=00000000000000000000:00000000000000011369, shard=3 master=00000000000000000000:00000000000000007584 target=00000000000000000000:00000000000000007443 2024-05-26T22:50:54.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:54.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:54.854 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:50:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:50:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:50:54.968 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:50:54.969 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:54.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011369", 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.222020Z" 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:50:54.971 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007443", 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:50:54.972 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.267505Z" 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:50:54Z" 2024-05-26T22:50:54.973 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:50:54.977 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000011651 target=00000000000000000000:00000000000000011369, shard=3 master=00000000000000000000:00000000000000007584 target=00000000000000000000:00000000000000007443 2024-05-26T22:50:59.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:59.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:50:59.979 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:00.093 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:51:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011651", 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:50:47.180399Z" 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007443", 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.267505Z" 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:51:00Z" 2024-05-26T22:51:00.098 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:51:00.099 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000007584 target=00000000000000000000:00000000000000007443 2024-05-26T22:51:05.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:05.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:05.102 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:05.215 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:51:05.216 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:05.217 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011651", 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:50:47.180399Z" 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:05.218 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007443", 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.267505Z" 2024-05-26T22:51:05.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:51:05Z" 2024-05-26T22:51:05.220 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:51:05.221 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000007584 target=00000000000000000000:00000000000000007443 2024-05-26T22:51:10.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.223 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:10.283 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.283 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: "instance_id": 2587703046061114392 2024-05-26T22:51:10.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000006152", 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.212993Z" 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011651", 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:50:47.180399Z" 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:51:10.340 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000011509", 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:44:14.689972Z" 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "incremental-sync", 2024-05-26T22:51:10.341 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000007584", 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:50:47.145967Z" 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 0, 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 0 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:51:10Z" 2024-05-26T22:51:10.342 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:51:10.343 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-05-26T22:51:10.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.389 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:10.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.439 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.489 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:51:10.490 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... ok 2024-05-26T22:51:10.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.490 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:10.541 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.541 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:51:10.588 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:48:46.190820Z" 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.590 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:51:10.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:51:10.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.592 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:10.648 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.681 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:10.681+0000 7f65fc32da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:10.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:10.682+0000 7f65fc32da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:10Z" 2024-05-26T22:51:10.704 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:10.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:51:10.705 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:51:10.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:10.705 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.757 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:51:10.788+0000 7f2d0ff9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:10.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:51:10.789+0000 7f2d0ff9aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:51:10Z" 2024-05-26T22:51:10.808 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:51:10.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:51:10.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.810 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.856 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:10.903 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:51:10.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:10.904 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:10.955 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.005 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:51:11.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.006 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.109 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:51:11.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.109 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:11.163 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.163 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.211 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:51:11.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.212 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:51:11.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.311 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:11.360 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.360 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 11, 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:51:11.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:51:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:51:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:51:11.415 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:51:11.416 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:51:11.417 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.418 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:51:11.419 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:51:11.428 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.428 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:51:11.429 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:51:11.430 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.431 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:51:11.432 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.433 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:51:11.434 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:51:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:51:11.436 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:51:11.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:51:11.437 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:51:11.492 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.492 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:51:11.585 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-84 2024-05-26T22:51:11.695 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xmqxus-85 2024-05-26T22:51:11.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:11.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:11.830 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:11.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:11.904 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:11.971 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:51:11.996 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:51:11.997 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:51:11.743013Z" 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:51:11.998 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:51:11.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:51:11.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:11.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:12.000 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:12.049 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:12.049 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:12.079 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:12.079+0000 7fb35e4e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:12.079 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:12.080+0000 7fb35e4e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:12.093 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:12.093 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:12.093 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:12.094 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:12.095 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:12.096 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:12.097 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:12Z" 2024-05-26T22:51:12.098 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:12.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:17.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:17.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:17.100 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:17.157 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:17.158 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:17.190 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:17.190+0000 7fc934e5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:17.191 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:17.192+0000 7fc934e5ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:17.206 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:17.207 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:17.209 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:17.210 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:17Z" 2024-05-26T22:51:17.211 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:17.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:22.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:22.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:22.213 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:22.270 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:22.271 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:22.303 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:22.303+0000 7f03b83dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:22.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:22.304+0000 7f03b83dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:22.318 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:22.318 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:22.318 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:22.320 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:22.321 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:22.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:22Z" 2024-05-26T22:51:22.323 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:22.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:27.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:27.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:27.326 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:27.383 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:27.383 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:27.415 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:27.416+0000 7fab2b9c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:27.416 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:27.417+0000 7fab2b9c6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:27.431 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:27.432 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:27.434 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:27.435 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:27Z" 2024-05-26T22:51:27.436 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:27.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:32.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:32.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:32.439 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:32.496 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:32.496 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:32.528 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:32.528+0000 7fe8108f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:32.529 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:32.529+0000 7fe8108f5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:32.543 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:32.543 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:32.543 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:32.543 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:32.544 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:32.545 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:32.547 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:32.548 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:32.549 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:32Z" 2024-05-26T22:51:32.550 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:32.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:37.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:37.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:37.552 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:37.609 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:37.610 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:37.642 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:37.642+0000 7f76111cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:37.643 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:37.643+0000 7f76111cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:37.661 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:37.661 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:37.661 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:37.661 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:37.661 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:37.662 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:37.663 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:37.664 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:37.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:37.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:37.665 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:37.666 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:37.666 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:37.666 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:37.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:37.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:37.667 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:37.670 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.671 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:37.672 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:37Z" 2024-05-26T22:51:37.672 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:37.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:42.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:42.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:42.675 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:42.733 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:42.733 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:42.767 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:42.768+0000 7f837de30a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:42.768 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:42.769+0000 7f837de30a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:42.783 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:42.783 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:42.783 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:42.783 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:42.783 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:42.784 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:42.785 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:42.786 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:42.787 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:42Z" 2024-05-26T22:51:42.788 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:42.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:47.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:47.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:47.791 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:47.848 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:47.849 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:47.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:47.884+0000 7f7df481da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:47.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:47.885+0000 7f7df481da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:47.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:47.900 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:47.901 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:47.902 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:47.903 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:47.904 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:47Z" 2024-05-26T22:51:47.904 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:47.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:52.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:52.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:52.907 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:52.961 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:52.961 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:52.993 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:52.994+0000 7f6052994a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:52.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:52.995+0000 7f6052994a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:53.008 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:53.008 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:53.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:53.010 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:53.011 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:53.012 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:53.013 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:53.014 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:53.015 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:52Z" 2024-05-26T22:51:53.015 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:53.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:51:58.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:58.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:51:58.018 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:51:58.075 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:51:58.075 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:51:58.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:58.110+0000 7fe00194fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:58.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:51:58.111+0000 7fe00194fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:51:58.124 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:51:58.125 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:58.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:58.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.128 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:51:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:51:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:51:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:51:58.129 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:51:58Z" 2024-05-26T22:51:58.129 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:51:58.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:03.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:03.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:03.131 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:03.187 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:03.188 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:03.220 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:03.221+0000 7f66812dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:03.221 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:03.222+0000 7f66812dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:03.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:03.238 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:03.240 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:03Z" 2024-05-26T22:52:03.241 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:03.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:08.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:08.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:08.244 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:08.300 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:08.301 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:08.334 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:08.334+0000 7fdcf1961a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:08.334 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:08.335+0000 7fdcf1961a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:08.349 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:08Z" 2024-05-26T22:52:08.354 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:08.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:13.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:13.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:13.357 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:13.413 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:13.413 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:13.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:13.447+0000 7ffa77e6aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:13.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:13.448+0000 7ffa77e6aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.462 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:13.463 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:13.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:13.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:13Z" 2024-05-26T22:52:13.467 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:13.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:18.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:18.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:18.470 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:18.526 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:18.527 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:18.559 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:18.559+0000 7f6dbac5ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:18.560 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:18.560+0000 7f6dbac5ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.575 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:18.577 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:18.578 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:18.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:18Z" 2024-05-26T22:52:18.580 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:18.581 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:23.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:23.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:23.583 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:23.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:23.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:23.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:23.676+0000 7f694c82da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:23.677+0000 7f694c82da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:23.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:23.697 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:23.698 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.699 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:23.700 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:23.700 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:23.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:23.700 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:23Z" 2024-05-26T22:52:23.700 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:23.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:28.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:28.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:28.703 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:28.761 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:28.762 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:28.794 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:28.795+0000 7f4ee12d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:28.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:28.795+0000 7f4ee12d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:28.810 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:28.810 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:28.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:28.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:28Z" 2024-05-26T22:52:28.815 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:28.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:33.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:33.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:33.818 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:33.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:33.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:33.908+0000 7fa300846a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:33.909+0000 7fa300846a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:33.923 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:33.923 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:33.924 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:33.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:33.926 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:33.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:33Z" 2024-05-26T22:52:33.928 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:33.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:38.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:38.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:38.930 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:38.987 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:38.987 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:39.019 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:39.021+0000 7f596172ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:39.020 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:39.022+0000 7f596172ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:39Z" 2024-05-26T22:52:39.040 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:39.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:44.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:44.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:44.043 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:44.101 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:44.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:44.135 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:44.136+0000 7f5584eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:44.135 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:44.137+0000 7f5584eb7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.150 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:44.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:44.153 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:44.154 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:44Z" 2024-05-26T22:52:44.155 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:44.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:49.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:49.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:49.158 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:49.215 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:49.215 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:49.249 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:49.250+0000 7f5137d7fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:49.249 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:49.251+0000 7f5137d7fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:49.264 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:49.264 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:49.264 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:49.264 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:49.264 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:49.265 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:49.266 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763726.190820_1376.1", 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:48:46.190820Z", 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:49.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:49Z" 2024-05-26T22:52:49.269 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:49.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763726.190820_1376.1 2024-05-26T22:52:54.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:54.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:54.271 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:54.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:54.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:54.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:54.362+0000 7fe4213efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:54.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:54.363+0000 7fe4213efa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.376 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:54.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:54.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763871.602498_1390.1", 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.602498Z", 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:54.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:54Z" 2024-05-26T22:52:54.381 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:54.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763871.602498_1390.1 2024-05-26T22:52:59.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:59.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:52:59.384 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:52:59.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:52:59.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:52:59.474 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:59.474+0000 7f6bf2d73a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:59.475 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:52:59.475+0000 7f6bf2d73a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:59.490 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:59.492 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:52:59.492 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:59.492 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:59.492 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:52:59.492 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763871.602498_1390.1", 2024-05-26T22:52:59.493 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.602498Z", 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:52:59.494 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:52:59.495 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:52:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:52:59Z" 2024-05-26T22:52:59.497 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:52:59.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763871.743013_1404.1 target=1_1716763871.602498_1390.1 2024-05-26T22:53:04.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.500 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:04.556 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:04.556 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:04.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:04.599+0000 7f10a9910a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:04.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:04.600+0000 7f10a9910a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:04.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:04.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.618 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.743013Z", 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.619 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.620 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:04Z" 2024-05-26T22:53:04.621 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:04.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:53:04.622 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:53:04.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.623 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:04.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:04.720+0000 7ff46f633a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:04.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:04.721+0000 7ff46f633a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:04.736 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:53:04.736 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:53:04.736 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:04.737 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:04.739 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:51:11.743013Z", 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:04.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:53:04Z" 2024-05-26T22:53:04.741 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:53:04.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:53:04.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.743 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:04.846 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:53:04.847 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:53:04.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:04.848 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:04.901 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:04.901 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:04.948 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024669", 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:47.116014Z" 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019620", 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.100962Z" 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:04Z" 2024-05-26T22:53:04.953 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:04.954 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024951 target=00000000000000000000:00000000000000024669, shard=3 master=00000000000000000000:00000000000000019761 target=00000000000000000000:00000000000000019620 2024-05-26T22:53:09.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:09.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:09.955 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:10.014 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:10.014 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.064 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:10.065 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024669", 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:47.116014Z" 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:10.066 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019620", 2024-05-26T22:53:10.067 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.100962Z" 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:10Z" 2024-05-26T22:53:10.068 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:10.069 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024951 target=00000000000000000000:00000000000000024669, shard=3 master=00000000000000000000:00000000000000019761 target=00000000000000000000:00000000000000019620 2024-05-26T22:53:15.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.071 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.131 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.131 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.180 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.181 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:15.182 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.183 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.184 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:15Z" 2024-05-26T22:53:15.185 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:15.186 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:53:15.193 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-05-26T22:53:15.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.194 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:51:11.743013Z" 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.305 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:53:15.306 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:53:15.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.307 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.357 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.357 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:15.388+0000 7ff48cfc7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:15.389+0000 7ff48cfc7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:15.402 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.403 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.404 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.743013Z", 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:15Z" 2024-05-26T22:53:15.407 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:15.408 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:53:15.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:53:15.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.409 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.462 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.462 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:15.494+0000 7f943b76aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:15.495+0000 7f943b76aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.509 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:53:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763871.743013_1404.1", 2024-05-26T22:53:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:51:11.743013Z", 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:53:15Z" 2024-05-26T22:53:15.514 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:53:15.515 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:53:15.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:15.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:15.516 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:15.562 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.611 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:53:15.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:15.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:15.612 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:53:15.705 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-86 2024-05-26T22:53:15.725 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-87 2024-05-26T22:53:15.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.742 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.790 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.791 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.826 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:53:15.845 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:53:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:15.845 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:15.741091Z" 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:15.846 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:53:15.847 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:15.847 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:53:15.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:53:15.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:15.848 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:15.899 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:15.899 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:15.930 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:15.930+0000 7fad44cdda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.931 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:15.931+0000 7fad44cdda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.718228_1409.1", 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.718228Z", 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:53:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:53:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:15Z" 2024-05-26T22:53:15.950 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:15.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716763995.741091_1416.1 target=1_1716763995.718228_1409.1 2024-05-26T22:53:20.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:20.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:20.952 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:21.008 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.009 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:21.042+0000 7f66d65dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:21.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:21.043+0000 7f66d65dea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:21.057 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:21.057 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:21.057 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:21.057 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:21.058 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:21.059 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:21.060 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:21.061 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:21Z" 2024-05-26T22:53:21.062 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:21.063 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:53:21.063 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:53:21.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:21.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:21.064 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:21.122 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.122 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:21.158+0000 7f0fe4483a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:21.159+0000 7f0fe4483a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:21.174 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:53:21.174 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:53:21.174 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:53:21.174 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:53:21.174 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:53:21.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:21.176 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.178 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:53:21Z" 2024-05-26T22:53:21.179 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:53:21.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:53:21.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.181 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.232 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.232 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.280 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:53:21.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.281 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.336 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.337 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.389 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:53:21.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=xmqxus-86', '--dest-bucket=xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=xmqxus-86', '--dest-bucket=xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.390 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --source-bucket=xmqxus-86 --dest-bucket=xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.446 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.495 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:53:21.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.496 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.596 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:53:21.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.596 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.642 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.689 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:53:21.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.690 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.739 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.739 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.790 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 12, 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:53:21.791 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:53:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:53:21.794 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.795 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.796 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:53:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:53:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:53:21.804 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:53:21.804 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:53:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-86", 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-86", 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:53:21.806 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:53:21.808 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:53:21.809 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:53:21.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:53:21.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:53:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:53:21.813 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:21.813 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:53:21.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:53:21.814 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:53:21.868 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:21.868 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:21.936 INFO:teuthology.orchestra.run.smithi007.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"test-zone2","bucket":"xmqxus-86"},"dest":{"zone":"test-zone1","bucket":"xmqxus-86"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"test-zone1","bucket":"xmqxus-86"},"dest":{"zone":"test-zone2","bucket":"xmqxus-86"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-05-26T22:53:22.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.083 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:22.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:22.259 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:22.299 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:15.741091Z" 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.300 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:53:22.301 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:53:22.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.302 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:22.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:22.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:22.382 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:22.382+0000 7fdd4a986a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:22.382 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:53:22.383+0000 7fdd4a986a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:22.396 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.397 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:22.398 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:22.399 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.400 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:22Z" 2024-05-26T22:53:22.401 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:22.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:53:22.403 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:53:22.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.403 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:22.481 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:22.481 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:22.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:22.553+0000 7fd9bf7cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:22.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:53:22.554+0000 7fd9bf7cba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:53:22.581 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:22.582 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:53:22.583 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:22.584 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:53:22.585 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:53:22Z" 2024-05-26T22:53:22.586 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:53:22.587 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:53:22.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.588 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:22.651 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:22.651 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:22.718 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000025233", 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.035722Z" 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:53:22.719 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:53:22.720 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:53:22.721 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:53:22.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:22.721 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:22.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:22.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:22.825 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:22.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:22.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:22.829 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:22.830 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:22.831 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:22Z" 2024-05-26T22:53:22.832 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:22.833 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:27.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:27.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:27.834 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:27.893 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:27.893 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:27.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:27.945 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:27.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:27.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:27Z" 2024-05-26T22:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:27.949 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:32.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:32.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:32.951 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:33.011 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:33.012 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:33.063 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:33.064 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:33.065 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:33.066 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:33Z" 2024-05-26T22:53:33.067 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:33.068 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:38.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:38.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:38.070 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:38.129 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:38.129 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:38.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:38.181 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:38.182 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:38.183 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:38.184 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:38Z" 2024-05-26T22:53:38.185 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:38.186 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:43.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:43.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:43.188 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:43.243 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:43.243 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:43.292 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:43Z" 2024-05-26T22:53:43.297 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:43.298 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:48.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:48.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:48.299 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:48.357 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:48.357 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:48.407 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:48.407 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:48.408 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:48.409 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:48.410 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:48.411 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:48.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:48.413 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:48Z" 2024-05-26T22:53:48.413 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:48.414 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:53.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:53.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:53.415 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:53.473 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:53.474 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:53.525 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:53.526 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:53.527 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:53.529 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:53.530 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:53.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:53.530 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:53Z" 2024-05-26T22:53:53.530 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:53.531 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:53:58.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:58.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:53:58.533 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:53:58.592 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:53:58.592 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:53:58.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:58.644 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:58.645 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:53:58.646 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:53:58Z" 2024-05-26T22:53:58.647 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:53:58.648 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:03.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:03.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:03.650 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:03.708 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:03.708 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:03.760 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:03.761 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:03.762 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:03.763 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:03.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:03.765 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:03Z" 2024-05-26T22:54:03.765 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:03.766 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:08.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:08.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:08.767 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:08.827 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:08.827 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.879 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:08.881 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:08.882 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:08.883 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:08.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:08.884 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:08Z" 2024-05-26T22:54:08.884 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:08.885 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:13.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:13.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:13.886 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:13.945 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:13.945 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:13.996 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:13.996 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:13.996 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:13.996 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:13.997 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:13.998 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:13.999 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:14.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:13Z" 2024-05-26T22:54:14.001 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:14.003 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:19.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:19.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:19.004 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:19Z" 2024-05-26T22:54:19.120 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:19.121 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:24.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:24.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:24.122 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:24.180 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:24.181 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:24.231 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:24.232 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:24.233 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:24.234 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:24.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:24Z" 2024-05-26T22:54:24.236 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:24.237 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:29.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:29.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:29.238 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:29.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:29.298 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:29Z" 2024-05-26T22:54:29.354 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:29.355 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:34.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:34.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:34.356 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:34.415 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:34.415 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:34.466 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:34.466 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:34.466 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:34.466 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:34.467 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:34.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.470 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:34.471 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:34.471 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:34.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:34.471 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:34Z" 2024-05-26T22:54:34.471 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:34.472 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:39.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:39.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:39.474 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:39.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:39.534 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:39.584 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:39.585 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:39.586 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:39.587 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:39.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:39Z" 2024-05-26T22:54:39.589 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:39.590 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:44.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:44.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:44.592 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:44.651 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:44.651 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:44.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:44.704 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:44.705 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:44.706 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:44Z" 2024-05-26T22:54:44.707 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:44.708 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:49.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:49.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:49.709 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:49.768 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:49.768 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:49.820 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:49.822 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:49.823 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:49Z" 2024-05-26T22:54:49.824 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:49.825 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:54.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:54.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:54.826 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:54:54.885 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:54:54.885 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:54:54.937 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:54:54.937 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:54:54.937 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:54:54.937 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:54:54.937 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:54:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:54:54.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:54:54.941 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:54:54.942 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:54:54.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:54:54.942 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:54:54Z" 2024-05-26T22:54:54.942 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:54:54.943 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:54:59.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:59.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:54:59.944 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:00.004 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:00.005 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:55:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:00Z" 2024-05-26T22:55:00.061 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:00.062 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:55:05.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:05.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:05.063 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:05.123 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:05.123 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:05.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:05.175 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:05.176 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:05.177 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:05.179 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:05Z" 2024-05-26T22:55:05.179 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:05.180 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:55:10.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:10.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:10.182 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:10.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:10.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:10.294 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:10.294 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:10.294 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:10.294 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:10.294 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:10.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:10.297 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:10.298 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:10.299 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:10.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:10.299 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:10Z" 2024-05-26T22:55:10.299 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:10.300 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:55:15.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:15.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:15.302 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:15.361 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:15.362 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:15.413 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:15.413 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:15.413 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:15.413 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:15.414 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:15.414 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:15.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.414 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:15.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:15.415 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:15.415 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:15.415 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:15.415 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:15.416 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:15.416 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:15.416 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:15.416 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:15.416 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:15.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:15.417 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:15.417 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:15.418 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:15.418 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:55:15.418 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:15.418 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:55:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:55:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:55:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:55:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:15.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:15.425 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:15Z" 2024-05-26T22:55:15.425 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:15.426 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:55:20.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:20.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:20.428 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:20.488 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:20.488 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:20.539 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:20.539 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:20.539 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000024951", 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.694038Z" 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022825", 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:38:46.457649Z" 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019761", 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:51:11.648009Z" 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:20Z" 2024-05-26T22:55:20.544 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:20.545 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025233 target=00000000000000000000:00000000000000024951, shard=2 master=00000000000000000000:00000000000000022966 target=00000000000000000000:00000000000000022825, shard=3 master=00000000000000000000:00000000000000019902 target=00000000000000000000:00000000000000019761 2024-05-26T22:55:25.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:25.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:25.547 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:25.606 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:25.606 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:25.657 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:25.658 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:25.659 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025233", 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.035722Z" 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:55:25.660 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:25.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:25Z" 2024-05-26T22:55:25.662 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:25.663 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:55:25.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.687 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:25.801 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:55:25.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.802 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:25.857 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:25.857 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:25.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:55:25.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:25.908 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.006 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:55:26.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.007 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.056 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.056 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.108 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:55:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 13, 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:55:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:55:26.110 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:55:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.112 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:55:26.113 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.115 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:55:26.116 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.118 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:55:26.118 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:55:26.118 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:55:26.118 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:55:26.118 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:55:26.119 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:55:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:55:26.122 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.123 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:55:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:55:26.124 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:55:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.126 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.127 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:55:26.129 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:55:26.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "allowed" 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:55:26.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.135 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:55:26.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:55:26.137 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:55:26.138 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:55:26.139 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:55:26.140 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:55:26.141 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:26.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:26.141 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:55:26.141 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:55:26.142 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:55:26.142 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:55:26.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.143 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:55:26.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.278 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.473 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:55:26.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.474 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:55:26.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.638 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.712 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.798 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-05-26T22:55:26.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.800 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.867 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.867 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:26.920 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T22:55:26.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:55:26.921 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:55:26.970 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:26.970 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:27.017 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T22:55:27.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:27.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:27.083 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:27.132 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:27.132 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:27.165 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:55:27.179 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:55:27.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:27.179 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:55:26.906544Z" 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:27.180 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:55:27.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:55:27.181 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:55:27.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:55:27.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:27.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:27.182 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:27.234 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:27.235 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:27.266 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:27.267+0000 7f89ca0a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:27.267 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:27.268+0000 7f89ca0a7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:27.286 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:27.286 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:27.286 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:27.286 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:27.286 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:27.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:27.289 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:27.290 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:27Z" 2024-05-26T22:55:27.291 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:27.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:32.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:32.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:32.294 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:32.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:32.389+0000 7fe3e6485a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:32.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:32.390+0000 7fe3e6485a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:32.403 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:32Z" 2024-05-26T22:55:32.409 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:32.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:37.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:37.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:37.411 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:37.502+0000 7faa02cc3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:37.503+0000 7faa02cc3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:37.517 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:37.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:37.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:37Z" 2024-05-26T22:55:37.524 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:37.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:42.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:42.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:42.526 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:42.584 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:42.584 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:42.620+0000 7f4d33231a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:42.621 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:42.621+0000 7f4d33231a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:42.634 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:42.635 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:42.636 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:42.637 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:42.638 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:42.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:42.640 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:42Z" 2024-05-26T22:55:42.640 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:42.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:47.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:47.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:47.642 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:47.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:47.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:47.734 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:47.733+0000 7fa1288a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:47.734 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:47.734+0000 7fa1288a5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:47Z" 2024-05-26T22:55:47.754 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:47.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:52.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:52.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:52.757 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:52.815 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:52.815 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:52.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:52.848+0000 7f552feb5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:52.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:52.849+0000 7f552feb5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:52.866 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:52.867 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:52.868 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:52Z" 2024-05-26T22:55:52.869 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:52.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:55:57.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:57.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:55:57.872 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:55:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:55:57.929 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:55:57.962 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:57.962+0000 7fbc5e80fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:57.963 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:55:57.963+0000 7fbc5e80fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:55:57.978 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:57.979 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:55:57.980 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:57.980 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:55:57.980 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:57.981 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:55:57.982 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:55:57.983 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:55:57.984 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:55:57Z" 2024-05-26T22:55:57.985 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:55:57.985 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:02.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:02.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:02.987 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:03.050 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:03.051 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:03.085 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:03.085+0000 7fa0e2fbca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:03.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:03.086+0000 7fa0e2fbca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:03.102 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:03.103 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:03.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:03.104 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:03.104 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:03.104 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:03.105 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:03.106 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:03.107 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:03.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:03.107 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:03Z" 2024-05-26T22:56:03.107 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:03.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:08.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:08.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:08.110 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:08.166 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:08.167 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:08.201 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:08.201+0000 7fe8d0616a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:08.202 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:08.202+0000 7fe8d0616a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:08.217 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:08.217 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:08.217 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:08.217 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:08.217 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:08.218 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:08.219 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:08.220 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:08.221 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.222 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:08.223 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:08.223 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:08.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:08.223 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:08Z" 2024-05-26T22:56:08.223 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:08.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:13.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:13.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:13.225 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:13.282 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:13.282 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:13.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:13.316+0000 7f12f31a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:13.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:13.317+0000 7f12f31a4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:13.335 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:13.335 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:13Z" 2024-05-26T22:56:13.341 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:13.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:18.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:18.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:18.343 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:18.400 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:18.401 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:18.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:18.435+0000 7f1de691ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:18.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:18.436+0000 7f1de691ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:18.450 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:18.454 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:18.454 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:18.454 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:18.454 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:18.454 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:18.455 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:18.456 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:18.457 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:18.458 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:18.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:18Z" 2024-05-26T22:56:18.460 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:18.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:23.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:23.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:23.463 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:23.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:23.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:23.554 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:23.555+0000 7f864449ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:23.555 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:23.556+0000 7f864449ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:23Z" 2024-05-26T22:56:23.575 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:23.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:28.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:28.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:28.577 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:28.634 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:28.635 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:28.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:28.671+0000 7fd7af19ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:28.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:28.671+0000 7fd7af19ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:28.686 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:28.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:28.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:28.689 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:28.690 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:28.691 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:28Z" 2024-05-26T22:56:28.692 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:28.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:33.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:33.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:33.694 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:33.751 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:33.751 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:33.786 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:33.787+0000 7f593bef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:33.788+0000 7f593bef7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:33.804 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:33.805 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:33.806 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:33.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:33.807 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:33Z" 2024-05-26T22:56:33.807 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:33.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:38.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:38.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:38.809 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:38.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:38.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:38.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:38.901+0000 7fe3f9695a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:38.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:38.902+0000 7fe3f9695a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:38.915 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:38.917 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:38.918 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:38.919 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:38Z" 2024-05-26T22:56:38.920 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:38.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:43.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:43.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:43.923 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:43.980 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:43.980 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:44.015 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:44.015+0000 7f1c7dabba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:44.015 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:44.016+0000 7f1c7dabba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:44.030 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:44.032 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:44.033 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:44.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:44Z" 2024-05-26T22:56:44.035 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:44.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:49.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:49.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:49.038 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:49.096 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:49.096 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:49.130 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:49.131+0000 7f9f5f860a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:49.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:49.132+0000 7f9f5f860a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:49.146 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:49.147 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:49.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:49.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:49.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:49.151 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:49Z" 2024-05-26T22:56:49.152 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:49.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:54.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:54.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:54.154 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:54.210 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:54.210 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:54.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:54.244+0000 7f2294b80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:54.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:54.245+0000 7f2294b80a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:54.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:54.261 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:54.262 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:54.263 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:54.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:54.265 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:54Z" 2024-05-26T22:56:54.265 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:54.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:56:59.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:59.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:56:59.267 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:56:59.324 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:56:59.324 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:56:59.358 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:59.358+0000 7fbf1ed1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:59.359 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:56:59.359+0000 7fbf1ed1ca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:56:59.374 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:56:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:56:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:56:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:56:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:56:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:56:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:56:59Z" 2024-05-26T22:56:59.379 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:56:59.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:04.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:04.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:04.382 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:04.439 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:04.440 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:04.474 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:04.474+0000 7f0a96f1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:04.474 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:04.475+0000 7f0a96f1fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:04.489 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:04.490 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:04.491 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:04.492 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:04.493 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:04Z" 2024-05-26T22:57:04.494 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:04.495 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:09.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:09.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:09.497 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:09.553 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:09.553 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:09.587 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:09.587+0000 7f7f30f63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:09.587 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:09.588+0000 7f7f30f63a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:09.602 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:57:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:09Z" 2024-05-26T22:57:09.607 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:09.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:14.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:14.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:14.610 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:14.666 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:14.666 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:14.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:14.702+0000 7fbe5d196a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:14.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:14.703+0000 7fbe5d196a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.720 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:14.721 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:14.722 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:14.723 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:14.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:14.725 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:14Z" 2024-05-26T22:57:14.725 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:14.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:19.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:19.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:19.727 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:19.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:19.785 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:19.818 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:19.820+0000 7f499fcc9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:19.819 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:19.821+0000 7f499fcc9a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:19.835 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:19.835 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:19.835 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:19.835 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:19.836 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:19.837 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:19.838 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:19.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.841 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:19.842 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:19.842 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:19.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:19.842 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:19Z" 2024-05-26T22:57:19.842 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:19.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:24.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:24.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:24.844 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:24.901 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:24.901 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:24.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:24.940+0000 7ff65d73fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:24.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:24.940+0000 7ff65d73fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:24.954 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:24.954 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:24.954 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:24.955 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:24.956 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716763995.741091_1416.1", 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:24.957 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:15.741091Z", 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:24.958 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:24.959 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:24Z" 2024-05-26T22:57:24.960 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:24.960 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764126.906544_1453.1 target=1_1716763995.741091_1416.1 2024-05-26T22:57:29.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:29.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:29.962 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.020 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.020 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:30.054+0000 7f89a1ad6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.054 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:30.055+0000 7f89a1ad6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:30.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.070 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:30.071 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.072 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:30.073 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:30.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.074 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:30Z" 2024-05-26T22:57:30.074 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:30.075 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:57:30.075 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:57:30.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.075 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.134 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.134 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:30.169+0000 7f95cb0a8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:30.170+0000 7f95cb0a8a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.185 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:30.185 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:57:30.185 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:57:30.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:30.187 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:30.188 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:30.190 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:30.191 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:57:30Z" 2024-05-26T22:57:30.193 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:30.195 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:57:30.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.196 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.247 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.247 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:57:30.302 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:57:30.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:57:30.304 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:57:30.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.304 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.358 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.358 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:57:30.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.408 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.409 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.411 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.412 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:30Z" 2024-05-26T22:57:30.413 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:30.413 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:57:30.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:30.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xmqxus-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:30.426 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xmqxus-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.531 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:57:30.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:30.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:30.532 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"sync_policy":{"groups":[]}} 2024-05-26T22:57:30.630 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2024-05-26T22:57:30.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.631 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.715 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:57:30.729 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:57:30.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.729 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:30.730 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:57:30.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.730 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:30.730 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764250.517651_1457.1", 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:57:30.517651Z" 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:30.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:30.732 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:30.732 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:57:30.732 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:30.732 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:57:30.733 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:57:30.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:30.734 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:30.785 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:30.786 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:30.815 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:30.816+0000 7f5976921a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.816 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:30.817+0000 7f5976921a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:30Z" 2024-05-26T22:57:30.835 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:30.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764250.517651_1457.1 target=1_1716764126.906544_1453.1 2024-05-26T22:57:35.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:35.837 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:35.894 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:35.895 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:35.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:35.928+0000 7f4be18a2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:35.928 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:35.929+0000 7f4be18a2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:35.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:35.945 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:35.946 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:35.947 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:35.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:35.949 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:35Z" 2024-05-26T22:57:35.949 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:35.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764250.517651_1457.1 target=1_1716764126.906544_1453.1 2024-05-26T22:57:40.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:40.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:40.951 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:41.009 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:41.009 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:41.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:41.043+0000 7fc5320c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:41.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:41.044+0000 7fc5320c4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:41.058 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:41.060 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:41.061 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:41.062 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.063 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:41.064 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:41.064 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:41.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:41.064 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:41Z" 2024-05-26T22:57:41.064 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:41.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764250.517651_1457.1 target=1_1716764126.906544_1453.1 2024-05-26T22:57:46.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:46.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:46.066 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:46.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:46.126 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:46.159 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:46.159+0000 7f7213212a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:46.160 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:46.160+0000 7f7213212a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:46.174 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:46.175 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:46.176 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:46.177 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764126.906544_1453.1", 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:26.906544Z", 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:46.178 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:46.179 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:46Z" 2024-05-26T22:57:46.180 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:46.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764250.517651_1457.1 target=1_1716764126.906544_1453.1 2024-05-26T22:57:51.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.182 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:51.240 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.240 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.274 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:51.274+0000 7ff3e296ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:51.276+0000 7ff3e296ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.290 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:51.290 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.291 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:51.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764250.517651_1457.1", 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:30.517651Z", 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.294 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:51Z" 2024-05-26T22:57:51.295 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:51.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:57:51.296 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:57:51.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.297 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:51.355 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.355 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:51.391+0000 7f45de31da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:51.392+0000 7f45de31da80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764250.517651_1457.1", 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:57:30.517651Z", 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:57:51Z" 2024-05-26T22:57:51.412 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:51.413 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:57:51.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:51.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:51.413 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.463 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.510 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[]} 2024-05-26T22:57:51.511 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-88 2024-05-26T22:57:51.531 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-89 2024-05-26T22:57:51.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.548 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:51.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.597 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:57:51.648 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:57:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:51.649 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:57:51.547074Z" 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.650 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:57:51.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:57:51.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.652 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:51.706 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.707 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:51.737+0000 7fdd7f48ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.738 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:51.738+0000 7fdd7f48ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:51.752 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:51.753 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.754 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.755 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:51.757 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:51.758 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:51.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:51.759 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:51Z" 2024-05-26T22:57:51.759 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:51.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:57:51.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:57:51.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:51.761 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:51.811 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:51.811 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:51.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:51.842+0000 7f19b5c02a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:51.843+0000 7f19b5c02a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:51.856 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.857 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:51.858 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:51.859 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:57:51.860 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764271.523759_1458.1", 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:57:51.523759Z", 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:51.862 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:57:51.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:51.864 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:57:51Z" 2024-05-26T22:57:51.864 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:51.865 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1716764271.547074_1465.1 target=1_1716764271.523759_1458.1 2024-05-26T22:57:56.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:56.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:56.866 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:56.925 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:56.926 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:56.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:56.962+0000 7f1cf4d8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:56.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:56.963+0000 7f1cf4d8ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.978 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:57:56.980 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:57:56.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:56.983 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:57:56Z" 2024-05-26T22:57:56.983 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:56.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:57:56.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:56.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:56.984 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.036 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.036 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.083 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:57:57.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.084 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.187 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:57:57.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-88', '--dest-bucket=xmqxus-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-88', '--dest-bucket=xmqxus-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.188 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-88 --dest-bucket=xmqxus-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.243 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.243 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.292 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-88","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:57:57.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.292 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.343 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.343 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.391 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-88","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:57:57.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.392 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 14, 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:57:57.493 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:57:57.494 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:57:57.495 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:57:57.496 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:57:57.497 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:57:57.498 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:57:57.499 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:57:57.500 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:57:57.501 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:57:57.507 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-88", 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-89", 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:57:57.508 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.509 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:57:57.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.511 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:57:57.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:57:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:57:57.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:57.515 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:57.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:57:57.517 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:57:57.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.564 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:57.621 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-88","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:57:57.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:57.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:57.859 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:57.932 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:57.933 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:57:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:57:51.547074Z" 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.055 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:57:58.056 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:57:58.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.056 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:58.113 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:58.113 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:58.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:58.148+0000 7f8a9d3fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:58.149 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:57:58.149+0000 7f8a9d3fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:58.164 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:58.164 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.165 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.166 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:57:58.167 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:58.168 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:58Z" 2024-05-26T22:57:58.169 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:58.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T22:57:58.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T22:57:58.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.171 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:58.247 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:58.248 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:58.303+0000 7ff2e43e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T22:57:58.305+0000 7ff2e43e4a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:57:58.325 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:57:58.325 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T22:57:58.325 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T22:57:58.325 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T22:57:58.325 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T22:57:58.326 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:57:58.328 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:57:58.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T22:57:58Z" 2024-05-26T22:57:58.330 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:57:58.331 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T22:57:58.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.332 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:58.396 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:58.396 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:58.459 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:57:58.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.459 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:57:58.459 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:57:58.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:57:57.808928Z" 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T22:57:58.460 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:57:57.828631Z" 2024-05-26T22:57:58.461 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:57:58.461 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:57:58.462 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:57:58.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:57:58.462 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:57:58.516 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:57:58.517 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.573 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.575 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:57:58.576 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:57:58.577 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:57:58Z" 2024-05-26T22:57:58.578 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:57:58.579 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:03.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:03.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:03.580 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:03.639 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:03.639 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:03.690 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:03.690 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:03.690 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:03.691 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:03.692 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:03.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:03.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:03Z" 2024-05-26T22:58:03.695 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:03.696 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:08.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:08.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:08.698 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:08.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:08.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:08.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:08.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:08.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:08.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:08.815 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:08Z" 2024-05-26T22:58:08.816 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:08.817 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:13.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:13.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:13.819 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:13.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:13.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:13.929 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:13.929 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:13.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:13.933 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:13Z" 2024-05-26T22:58:13.935 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:13.936 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:18.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:18.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:18.937 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:19.043 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:19Z" 2024-05-26T22:58:19.048 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:19.049 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:24.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:24.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:24.050 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:24.109 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:24.109 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:24.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:24.162 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:24.163 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:24.164 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:24.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:24.165 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:24Z" 2024-05-26T22:58:24.165 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:24.166 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:29.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:29.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:29.167 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:29.225 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:29.225 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:29.277 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:29.279 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:29.280 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:29.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:29Z" 2024-05-26T22:58:29.282 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:29.283 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:34.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:34.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:34.284 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:34.343 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:34.343 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:34.395 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:34.395 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:34.395 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:34.395 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:34.395 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:34.396 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.399 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:34.400 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:34.400 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:34.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:34.400 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:34Z" 2024-05-26T22:58:34.400 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:34.401 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:39.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:39.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:39.402 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:39.461 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:39.461 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:39.512 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:39.512 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:39.512 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:39.516 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:39.517 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:39.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:39.517 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:39Z" 2024-05-26T22:58:39.517 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:39.518 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:44.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:44.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:44.520 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:44.579 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:44.580 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.631 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:44.632 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:44.633 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:44.634 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:44.635 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:44Z" 2024-05-26T22:58:44.636 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:44.637 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:49.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:49.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:49.638 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:49.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:49.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:49.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:49.753 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:49.754 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:49.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:49Z" 2024-05-26T22:58:49.756 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:49.758 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:54.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:54.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:54.760 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:54.819 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:54.820 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:54.871 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:54Z" 2024-05-26T22:58:54.876 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:54.877 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:58:59.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:59.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:58:59.878 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:58:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:58:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:58:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:58:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:58:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:58:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:58:59Z" 2024-05-26T22:58:59.995 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:58:59.996 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:04.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:04.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:04.998 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:05.057 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:05.057 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:05.108 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:05.108 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:05.108 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:05.108 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:05.108 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:05.109 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:05.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:05.111 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:05.112 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:05.113 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:05.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:05.113 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:05Z" 2024-05-26T22:59:05.113 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:05.115 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:10.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:10.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:10.117 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:10.176 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:10.176 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.227 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:10.228 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:10.229 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:10.230 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:10Z" 2024-05-26T22:59:10.231 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:10.232 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:15.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:15.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:15.234 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:15.293 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:15.293 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:15.345 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:15.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:15.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:15.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:15.349 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:15Z" 2024-05-26T22:59:15.350 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:15.351 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:20.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:20.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:20.353 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:20.412 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:20.412 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:20.464 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:20.464 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:20.464 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:20.466 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:20.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:20.468 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:20Z" 2024-05-26T22:59:20.469 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:20.470 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:25.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:25.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:25.471 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:25.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:25.531 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:25.583 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:25.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025515", 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:25.585 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:55:27.042632Z" 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:25.586 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000019902", 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:21.994359Z" 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:25.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:25Z" 2024-05-26T22:59:25.588 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:25.589 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025656 target=00000000000000000000:00000000000000025515, shard=3 master=00000000000000000000:00000000000000020043 target=00000000000000000000:00000000000000019902 2024-05-26T22:59:30.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:30.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:30.591 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:30.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:30.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T22:59:30.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.808928Z" 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:30.704 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.828631Z" 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:30.705 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:30Z" 2024-05-26T22:59:30.706 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:30.707 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T22:59:30.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.728 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:30.843 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-05-26T22:59:30.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.843 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:30.947 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T22:59:30.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:30.949 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.050 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:59:31.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.051 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 15, 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T22:59:31.155 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:59:31.156 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:59:31.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:59:31.158 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:59:31.159 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T22:59:31.160 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T22:59:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T22:59:31.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.163 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T22:59:31.163 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:59:31.163 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T22:59:31.166 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.168 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "allowed" 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T22:59:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T22:59:31.170 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.171 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:59:31.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T22:59:31.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:31.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:31.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T22:59:31.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T22:59:31.174 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T22:59:31.174 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T22:59:31.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.175 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.291 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T22:59:31.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.292 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xmqxus-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.358 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.358 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.463 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T22:59:31.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.463 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket xmqxus-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.543 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.544 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.647 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T22:59:31.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-88', '--source-bucket=*', '--dest-bucket=xmqxus-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=xmqxus-88', '--source-bucket=*', '--dest-bucket=xmqxus-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.648 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=xmqxus-88 '--source-bucket=*' --dest-bucket=xmqxus-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.818 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-05-26T22:59:31.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.818 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xmqxus-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:31.954 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T22:59:31.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xmqxus-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T22:59:31.954 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xmqxus-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T22:59:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:32.059 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T22:59:32.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:32.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:32.060 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:32.109 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:32.109 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T22:59:32.155 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:32.156 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:59:31.936418Z" 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T22:59:32.157 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T22:59:32.158 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T22:59:32.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:32.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:32.158 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:32.208 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:32.208 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:32.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:32.238+0000 7fad223d3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:32.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:32.239+0000 7fad223d3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:32.252 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:32Z" 2024-05-26T22:59:32.257 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:32.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T22:59:37.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:37.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:37.260 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:37.351 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:37.351+0000 7f34dc4d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:37.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:37.352+0000 7f34dc4d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:37.367 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:37.368 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:37.369 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:37.370 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:37.371 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:37.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:37.373 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:37Z" 2024-05-26T22:59:37.373 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:37.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T22:59:42.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:42.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:42.375 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:42.432 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:42.432 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:42.467 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:42.467+0000 7fb13c5e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:42.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:42.468+0000 7fb13c5e1a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:42.483 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:42.484 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:42.485 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:42.486 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:42.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:42.488 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:42Z" 2024-05-26T22:59:42.488 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:42.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T22:59:47.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:47.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:47.490 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:47.547 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:47.547 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:47.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:47.581+0000 7f5229838a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:47.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:47.582+0000 7f5229838a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:47.596 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:47.596 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:47.596 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:47.597 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:47.598 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:47.599 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:47.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:47Z" 2024-05-26T22:59:47.601 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:47.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T22:59:52.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:52.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:52.603 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:52.660 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:52.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:52.694 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:52.694+0000 7fef44336a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:52.695 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:52.695+0000 7fef44336a80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:52.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:52.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:52.714 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:52Z" 2024-05-26T22:59:52.715 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:52.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T22:59:57.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:57.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T22:59:57.718 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T22:59:57.774 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T22:59:57.775 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T22:59:57.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:57.809+0000 7f1190ddda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:57.810 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T22:59:57.810+0000 7f1190ddda80 20 RGW-SYNC:meta: read sync status 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T22:59:57.825 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T22:59:57.826 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:57.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:57.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T22:59:57.829 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T22:59:57Z" 2024-05-26T22:59:57.830 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T22:59:57.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:02.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:02.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:02.833 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:02.883 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:02.883 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:02.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:02.913+0000 7fe86d6c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:02.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:02.913+0000 7fe86d6c2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:02.927 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:02.927 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:02.928 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:02.929 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:02.931 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:02Z" 2024-05-26T23:00:02.932 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:02.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:07.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:07.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:07.935 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:07.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:07.994 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:08.027 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:08.027+0000 7f87ee630a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:08.028 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:08.028+0000 7f87ee630a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:08.042 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:08.046 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:08.047 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:08Z" 2024-05-26T23:00:08.048 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:08.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:13.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:13.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:13.049 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:13.107 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:13.107 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:13.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:13.142+0000 7eff967f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:13.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:13.143+0000 7eff967f0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:13.157 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:13.157 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:13.157 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:13.157 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:13.158 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:13.159 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:13.160 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:13.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:13Z" 2024-05-26T23:00:13.162 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:13.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:18.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:18.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:18.165 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:18.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:18.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:18.255 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:18.256+0000 7fb5ac4b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:18.256 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:18.257+0000 7fb5ac4b7a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:18.270 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:18.271 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:18.272 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:18.273 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:18.274 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:18.275 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:18.276 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:18.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:18.276 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:18Z" 2024-05-26T23:00:18.276 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:18.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:23.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:23.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:23.278 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:23.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:23.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:23.370+0000 7fac36916a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:23.370 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:23.371+0000 7fac36916a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:23.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:23.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:23.390 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:23Z" 2024-05-26T23:00:23.390 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:23.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:28.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:28.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:28.392 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:28.448 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:28.449 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:28.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:28.483+0000 7fbc293cfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:28.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:28.484+0000 7fbc293cfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:28.498 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:28Z" 2024-05-26T23:00:28.503 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:28.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:33.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:33.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:33.506 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:33.563 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:33.563 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:33.597 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:33.597+0000 7faa131daa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:33.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:33.598+0000 7faa131daa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:33.612 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:33.613 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:33.614 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:33.615 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:33.616 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:33Z" 2024-05-26T23:00:33.617 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:33.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:38.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:38.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:38.620 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:38.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:38.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:38.712 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:38.711+0000 7f343128ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:38.713 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:38.712+0000 7f343128ba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:38.727 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.728 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:38.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:38.730 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:38.731 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:38.733 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:38Z" 2024-05-26T23:00:38.734 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:38.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:43.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:43.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:43.737 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:43.794 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:43.794 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:43.829 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:43.829+0000 7fb87e03fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:43.830 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:43.830+0000 7fb87e03fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:43.844 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:43.845 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:43.846 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:43.847 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:43.848 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:43Z" 2024-05-26T23:00:43.849 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:43.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:48.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:48.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:48.852 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:48.909 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:48.909 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:48.944 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:48.944+0000 7fc30e65ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:48.944 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:48.945+0000 7fc30e65ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:48.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:48.961 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:48.962 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:48.963 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:48.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:48.965 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:48Z" 2024-05-26T23:00:48.965 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:48.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:53.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:53.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:53.967 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:54.024 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:54.024 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:54.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:54.058+0000 7f8d5f803a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:54.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:54.059+0000 7f8d5f803a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.074 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:54.076 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:54.077 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:54.078 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:54Z" 2024-05-26T23:00:54.079 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:54.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:00:59.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:59.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:00:59.081 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:00:59.138 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:00:59.138 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:00:59.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:59.173+0000 7f9578afda80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:59.174 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:00:59.174+0000 7f9578afda80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:00:59.188 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:00:59.189 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:59.190 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:59.191 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:59.192 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:00:59Z" 2024-05-26T23:00:59.193 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:00:59.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:04.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:04.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:04.196 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:04.255 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:04.255 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:04.288 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:04.288+0000 7f38d0e39a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:04.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:04.289+0000 7f38d0e39a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:04.304 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:04.304 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:04.304 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:04.305 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:04.306 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:04.307 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:04.308 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:04.309 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:04Z" 2024-05-26T23:01:04.310 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:04.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:09.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:09.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:09.312 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:09.369 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:09.369 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:09.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:09.404+0000 7f93312cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:09.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:09.405+0000 7f93312cda80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:09Z" 2024-05-26T23:01:09.425 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:09.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:14.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:14.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:14.427 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:14.483 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:14.483 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:14.517+0000 7f6b88691a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:14.518+0000 7f6b88691a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:14.531 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:14Z" 2024-05-26T23:01:14.536 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:14.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:19.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:19.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:19.539 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:19.595 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:19.595 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:19.629 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:19.630+0000 7ff5da779a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:19.630 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:19.631+0000 7ff5da779a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:19.645 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:19.646 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:19.647 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:19.648 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:19.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:19.650 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:19Z" 2024-05-26T23:01:19.650 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:19.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:24.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:24.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:24.652 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:24.708 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:24.708 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:24.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:24.743+0000 7fa8cd812a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:24.744 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:24.744+0000 7fa8cd812a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:24.759 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:24.760 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:24.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:24.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:24.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:24Z" 2024-05-26T23:01:24.764 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:24.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:29.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:29.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:29.767 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:29.824 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:29.824 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:29.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:29.859+0000 7fe10f41fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:29.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:29.860+0000 7fe10f41fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:29.876 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:29.876 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:29.876 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:29.877 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:29.878 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764271.547074_1465.1", 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:29.879 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:51.547074Z", 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:29.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:29Z" 2024-05-26T23:01:29.881 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:29.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764371.936418_1495.1 target=1_1716764271.547074_1465.1 2024-05-26T23:01:34.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:34.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:34.884 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:34.939 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:34.939 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:34.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:34.972+0000 7fb9d14b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:34.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:34.973+0000 7fb9d14b3a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:34Z" 2024-05-26T23:01:34.993 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:34.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:01:34.994 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:01:34.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:34.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:34.994 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:35.088+0000 7f210b4d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:35.089+0000 7f210b4d0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.106 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:01:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:01:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:01:35.106 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.107 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.109 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:35.110 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:01:35Z" 2024-05-26T23:01:35.111 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:01:35.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:01:35.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.171 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.221 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.221 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.255 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:01:35.269 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:01:35.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.269 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:59:31.936418Z" 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:35.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:01:35.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.271 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:01:35.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:01:35.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.272 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.323 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.323 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.354 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:35.354+0000 7f28abc06a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.355 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:35.355+0000 7f28abc06a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:35.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.372 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:35.373 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:35.374 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:35.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.374 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:35Z" 2024-05-26T23:01:35.374 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:35.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:01:35.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:01:35.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.375 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:35.459+0000 7f993abeaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:35.460+0000 7f993abeaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:35.473 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:01:35.473 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.474 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.475 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.476 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:35.477 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:01:35Z" 2024-05-26T23:01:35.478 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:01:35.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:01:35.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.480 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:35.589 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:01:35.590 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:01:35.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:35.590 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:35.642 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:35.642 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:35.691 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.808928Z" 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.828631Z" 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:01:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:01:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:35Z" 2024-05-26T23:01:35.696 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:35.697 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025797 target=00000000000000000000:00000000000000025656, shard=3 master=00000000000000000000:00000000000000020184 target=00000000000000000000:00000000000000020043 2024-05-26T23:01:40.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:40.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:40.698 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:40.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:40.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:40.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:40.811 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.808928Z" 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:40.812 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T23:01:40.813 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.828631Z" 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:40.814 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:40Z" 2024-05-26T23:01:40.815 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:40.815 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025797 target=00000000000000000000:00000000000000025656, shard=3 master=00000000000000000000:00000000000000020184 target=00000000000000000000:00000000000000020043 2024-05-26T23:01:45.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:45.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:45.816 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:45.876 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:45.876 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:45.928 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:45.928 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:45.928 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:45.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:45.930 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.808928Z" 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:45.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.828631Z" 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.932 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:45.933 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:01:45.933 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:01:45.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:45.933 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:45Z" 2024-05-26T23:01:45.933 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:45.934 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025797 target=00000000000000000000:00000000000000025656, shard=3 master=00000000000000000000:00000000000000020184 target=00000000000000000000:00000000000000020043 2024-05-26T23:01:50.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:50.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:50.935 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:50.996 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:50.996 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:51.049 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025656", 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.808928Z" 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:51.050 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020043", 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:51.051 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:57:57.828631Z" 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:51Z" 2024-05-26T23:01:51.052 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:51.053 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025797 target=00000000000000000000:00000000000000025656, shard=3 master=00000000000000000000:00000000000000020184 target=00000000000000000000:00000000000000020043 2024-05-26T23:01:56.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.055 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.114 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.114 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.167 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:56.167 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:56.167 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:56.167 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:56.167 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.168 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:56.171 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:01:56.172 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:01:56.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.172 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:56Z" 2024-05-26T23:01:56.172 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:56.173 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:01:56.185 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2024-05-26T23:01:56.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.186 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.243 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.243 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.280 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:59:31.936418Z" 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.297 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:01:56.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:01:56.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.298 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.349 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.379 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:56.379+0000 7f1c8dcaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.379 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:56.380+0000 7f1c8dcaea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.393 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:56.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.395 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:56.396 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.397 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:56.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.399 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:56Z" 2024-05-26T23:01:56.399 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:56.400 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:01:56.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:01:56.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.400 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.453 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.453 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:56.488+0000 7f7a8e2a0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:01:56.489+0000 7f7a8e2a0a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.503 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:01:56.504 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:01:56.505 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:01:56.506 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764371.936418_1495.1", 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:59:31.936418Z", 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:01:56.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:01:56.508 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:01:56.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.509 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:01:56Z" 2024-05-26T23:01:56.509 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:01:56.510 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:01:56.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:01:56.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:01:56.510 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:01:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.558 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.609 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:01:56.610 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-90 2024-05-26T23:01:56.627 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-91 2024-05-26T23:01:56.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.643 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:01:56.738 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:01:56.642775Z" 2024-05-26T23:01:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:01:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:01:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:01:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:01:56.740 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:01:56.741 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:01:56.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:01:56.741 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:01:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:01:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:01:56.828 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:56.828+0000 7fa7f9ea2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.829 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:01:56.829+0000 7fa7f9ea2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:01:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.619951_1496.1", 2024-05-26T23:01:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.619951Z", 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:01:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:01:56Z" 2024-05-26T23:01:56.851 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:01:56.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764516.642775_1503.1 target=1_1716764516.619951_1496.1 2024-05-26T23:02:01.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:01.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:01.854 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:01.945 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:02:01.946+0000 7f7d2b9e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:01.946 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:02:01.947+0000 7f7d2b9e5a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:02:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:01Z" 2024-05-26T23:02:01.966 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:01.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:02:01.967 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:02:01.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:01.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:01.967 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:02.025 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:02:02.061+0000 7fa316628a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:02.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:02:02.063+0000 7fa316628a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:02:02.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:02:02.080 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:02.081 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:02:02Z" 2024-05-26T23:02:02.082 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:02:02.083 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:02:02.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.084 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.185 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:02:02.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.185 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.241 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.241 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:02:02.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-90', '--dest-bucket=xmqxus-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-90', '--dest-bucket=xmqxus-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.290 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xmqxus-90 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-90 --dest-bucket=xmqxus-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.345 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.346 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.395 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:02:02.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-91', '--dest-bucket=xmqxus-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-91', '--dest-bucket=xmqxus-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.396 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xmqxus-91 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-91 --dest-bucket=xmqxus-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.451 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.451 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:02:02.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.509 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.611 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:02:02.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.612 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.661 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.661 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.713 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:02:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 16, 2024-05-26T23:02:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T23:02:02.713 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.714 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:02:02.715 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T23:02:02.716 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:02:02.717 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:02:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T23:02:02.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T23:02:02.720 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T23:02:02.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.722 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:02:02.723 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-90", 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-90", 2024-05-26T23:02:02.724 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:02:02.725 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-91", 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:02:02.726 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:02:02.727 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:02:02.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T23:02:02.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.730 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T23:02:02.731 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.732 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:02.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:02:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T23:02:02.734 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:02.734 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:02:02.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:02:02.735 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:02:02.782 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:02.782 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:02.874 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:02:03.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.087 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:03.165 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:03.261 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.294 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:01:56.642775Z" 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.295 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:02:03.296 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:02:03.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.297 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:03.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:03.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:03.392 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:02:03.393+0000 7f45e7db2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:03.394 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:02:03.395+0000 7f45e7db2a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:03.415 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:03.417 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:02:03.419 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:02:03.420 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:02:03.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.421 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:03Z" 2024-05-26T23:02:03.421 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:03.422 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:02:03.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:02:03.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.423 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:03.493 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:03.493 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:03.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:02:03.546+0000 7fb1daa4aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:02:03.548+0000 7fb1daa4aa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:02:03.570 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:03.571 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:02:03.573 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:02:03.574 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:02:03.575 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:02:03.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.575 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:02:03Z" 2024-05-26T23:02:03.575 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:02:03.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:02:03.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.576 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:03.692 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:02:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:02:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:02:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000026079", 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:02:03.032555Z" 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000020325", 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:02:02.993968Z" 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:02:03.693 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:02:03.694 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:02:03.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:03.695 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:03.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:03.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.793 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.794 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:03.795 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:03.796 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:03Z" 2024-05-26T23:02:03.797 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:03.798 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:08.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:08.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:08.800 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:08.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:08.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.911 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:08.912 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:08.913 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:08.914 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:08Z" 2024-05-26T23:02:08.915 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:08.916 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:13.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:13.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:13.918 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:13.977 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:13.977 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:14.029 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:14.029 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:14.029 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:14.029 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:14.030 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:14.031 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:14.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:14.036 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:14Z" 2024-05-26T23:02:14.036 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:14.037 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:19.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:19.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:19.039 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:19.098 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:19.098 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:19Z" 2024-05-26T23:02:19.155 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:19.156 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:24.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:24.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:24.157 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:24.268 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:24.268 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:24.268 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:24.268 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:24.268 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:24.271 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:24.272 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:24.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:24.273 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:24Z" 2024-05-26T23:02:24.273 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:24.274 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:29.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:29.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:29.275 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:29.334 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:29.334 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:29.386 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:29Z" 2024-05-26T23:02:29.391 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:29.392 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:34.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:34.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:34.393 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:34.452 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:34.452 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:34.503 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:34.504 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:34.505 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:34.506 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:34.507 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:34Z" 2024-05-26T23:02:34.508 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:34.509 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:39.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:39.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:39.511 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:39.569 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:39.569 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:39.626 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:39Z" 2024-05-26T23:02:39.626 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:39.627 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:44.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:44.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:44.628 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:44.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:44.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:44.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:44.741 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:44.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:44.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:44Z" 2024-05-26T23:02:44.745 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:44.746 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:49.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:49.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:49.748 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:49.805 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:49.806 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:49.856 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:49.856 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:49.856 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:49.856 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:49.857 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:49.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:49.859 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:49.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:49.861 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:49Z" 2024-05-26T23:02:49.861 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:49.861 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:54.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:54.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:54.863 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:02:54.922 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:02:54.922 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:02:54.973 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:54.974 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:02:54.975 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:54.976 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:02:54Z" 2024-05-26T23:02:54.978 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:02:54.979 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:02:59.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:59.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:02:59.981 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:00.040 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:00.040 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:00.092 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:00.092 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:00.092 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:00.093 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:00.094 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:00.095 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:00.096 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:00Z" 2024-05-26T23:03:00.097 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:00.098 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:05.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:05.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:05.101 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:05.159 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:05.159 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:05.211 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:05.212 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:05.214 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:05.215 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:05.216 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:05.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:05.216 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:05Z" 2024-05-26T23:03:05.216 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:05.217 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:10.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:10.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:10.218 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:10.327 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:10.327 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:10.327 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:10.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:10.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.331 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:10.332 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:10.332 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:10.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:10.332 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:10Z" 2024-05-26T23:03:10.332 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:10.333 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:15.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:15.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:15.334 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:15.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:15.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:15.444 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:15.444 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:15.444 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:15.444 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:15.445 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:15.448 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:15Z" 2024-05-26T23:03:15.449 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:15.450 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:20.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:20.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:20.451 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:20.509 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:20.509 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:20Z" 2024-05-26T23:03:20.566 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:20.567 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:25.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:25.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:25.568 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:25.627 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:25.627 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.679 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:25.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:25.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:25.682 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:25.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:25Z" 2024-05-26T23:03:25.684 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:25.685 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:30.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:30.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:30.687 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:30.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:30.746 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:30.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014364", 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:50:17.103529Z" 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000025797", 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.125451Z" 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:30.804 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:30.805 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020184", 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:35.133378Z" 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:30.806 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:30Z" 2024-05-26T23:03:30.807 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:30.808 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014505 target=00000000000000000000:00000000000000014364, shard=1 master=00000000000000000000:00000000000000026079 target=00000000000000000000:00000000000000025797, shard=3 master=00000000000000000000:00000000000000020325 target=00000000000000000000:00000000000000020184 2024-05-26T23:03:35.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:35.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:35.809 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:35.868 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:35.868 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:35.921 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:35.922 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026079", 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.032555Z" 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:35.923 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020325", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:35.924 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:02.993968Z" 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:35Z" 2024-05-26T23:03:35.925 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:35.926 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:03:35.938 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-05-26T23:03:35.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:35.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:35.939 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:03:36.049 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:03:36.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.049 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:01:56.642775Z" 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.050 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:36.051 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:03:36.051 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.051 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:03:36.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:03:36.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.052 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.133 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:36.133+0000 7f8a44045a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:36.134+0000 7f8a44045a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.148 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:36.148 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:36.148 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:36.148 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:36.149 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.150 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.151 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.152 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:36Z" 2024-05-26T23:03:36.153 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:36.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:03:36.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:03:36.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.155 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:36.207 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:36.207 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:03:36.240+0000 7f6504befa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:03:36.241+0000 7f6504befa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:03:36.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:03:36.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:03:36Z" 2024-05-26T23:03:36.260 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:03:36.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:03:36.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:03:36.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:03:36.261 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:03:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.356 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:03:36.356 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-92 2024-05-26T23:03:36.376 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xmqxus-93 2024-05-26T23:03:36.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.394 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.476 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:03:36.490 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:03:36.393027Z" 2024-05-26T23:03:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:03:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:03:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:03:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:03:36.492 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:03:36.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:03:36.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:36.493 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:36.543 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:36.543 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:36.573 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:36.573+0000 7f9572a6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.574 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:36.574+0000 7f9572a6da80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:36.588 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:36.588 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:36.588 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.589 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:36.590 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.591 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:36.592 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:36.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:36Z" 2024-05-26T23:03:36.594 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:36.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:03:41.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:41.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:41.597 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:41.654 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:41.654 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:41.688 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:41.688+0000 7f049b692a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:41.689 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:41.689+0000 7f049b692a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:41.704 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:41.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:41.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.708 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:41.709 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:41.709 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:41.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:41.709 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:41Z" 2024-05-26T23:03:41.709 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:41.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:03:46.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:46.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:46.711 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:46.768 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:46.768 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:46.805 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:46.805+0000 7f685000fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:46.806 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:46.806+0000 7f685000fa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:46.821 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:46.821 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:46.821 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:46.821 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:46.822 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:46.823 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:46.824 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:46.825 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:46Z" 2024-05-26T23:03:46.826 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:46.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:03:51.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:51.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:51.829 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:51.886 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:51.886 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:51.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:51.919+0000 7f2a0b3fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:51.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:51.920+0000 7f2a0b3fca80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:51.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:51.937 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:51.938 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:51.939 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:51.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:51.941 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:51Z" 2024-05-26T23:03:51.941 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:51.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:03:56.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:56.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:03:56.943 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:03:57.001 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:03:57.001 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:03:57.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:57.036+0000 7f6cc3686a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:57.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:03:57.037+0000 7f6cc3686a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:03:57.051 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:03:57.051 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:03:57.051 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:03:57.051 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:57.052 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:57.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:57.054 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:03:57.055 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:03:57Z" 2024-05-26T23:03:57.056 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:03:57.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:04:02.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:02.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:02.058 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:02.116 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:02.116 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:02.152 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:02.152+0000 7ffb19ccea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:02.153 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:02.154+0000 7ffb19ccea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:02.171 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:04:02.172 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:02.173 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764516.642775_1503.1", 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:01:56.642775Z", 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:02.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:02.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:02Z" 2024-05-26T23:04:02.176 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:02.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764616.393027_1523.1 target=1_1716764516.642775_1503.1 2024-05-26T23:04:07.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:07.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:07.178 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:07.235 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.235 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:07.273+0000 7ff4cea66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:07.273 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:07.274+0000 7ff4cea66a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:07.288 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:04:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:04:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:07Z" 2024-05-26T23:04:07.293 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:07.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:04:07.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:04:07.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:07.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:07.295 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:07.353 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.353 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:04:07.390+0000 7f59b2e5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:07.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:04:07.391+0000 7f59b2e5da80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:07.405 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:04:07.405 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:04:07.405 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:04:07.405 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:07.406 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:07.407 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:04:07.408 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:07.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:04:07Z" 2024-05-26T23:04:07.410 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:04:07.411 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:04:07.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.412 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.513 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:04:07.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.514 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.619 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:04:07.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.620 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xmqxus-92 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-92 --dest-bucket=xmqxus-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.675 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.725 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:04:07.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xmqxus-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.725 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xmqxus-93 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-92 --dest-bucket=xmqxus-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.781 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.830 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:04:07.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=xmqxus-92', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.830 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=xmqxus-92 --dest-bucket=xmqxus-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.886 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.887 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:07.936 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-92","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:04:07.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:07.936 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.036 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-92","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:04:08.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:08.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:08.037 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:08.086 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:08.086 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 17, 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:08.138 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:04:08.139 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:04:08.140 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:04:08.142 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:04:08.143 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.144 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:04:08.145 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.146 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T23:04:08.147 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T23:04:08.148 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.149 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T23:04:08.150 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:04:08.151 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-93", 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.152 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-90", 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-90", 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-91", 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.157 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-92", 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-93", 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-93", 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:04:08.161 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "enabled" 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.162 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.165 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T23:04:08.165 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:04:08.165 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T23:04:08.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:04:08.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:04:08.167 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.168 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:04:08.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:08.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:04:08.169 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:04:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xmqxus-92","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:04:08.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.407 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:08.478 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:08.478 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.581 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:04:08.630 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:04:08.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.630 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:03:36.393027Z" 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:04:08.632 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.633 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:04:08.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:04:08.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.635 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:08.737+0000 7fd5deb28a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:04:08.739+0000 7fd5deb28a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:08.754 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:08.754 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:08.754 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:08.754 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:08.754 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:08.755 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:08.756 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:08.757 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:04:08.758 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:08.759 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:08.760 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:04:08.761 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:08.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:08.763 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:08Z" 2024-05-26T23:04:08.764 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:08.765 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:04:08.765 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:04:08.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:08.840 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:08.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:04:08.924+0000 7f80092aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:04:08.926+0000 7f80092aaa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:04:08.947 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:04:08.947 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.948 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:04:08.949 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:04:08.950 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:08.952 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:04:08.953 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:04:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:08.955 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:04:08Z" 2024-05-26T23:04:08.956 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:04:08.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:04:08.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:08.958 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:09.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:04:09.101 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:04:09.102 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:04:09.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:09.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:09.103 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:09.154 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:09.154 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:09.201 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:09.203 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:09Z" 2024-05-26T23:04:09.205 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:09.206 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:14.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:14.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:14.208 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:14.268 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:14.268 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:14.320 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:14Z" 2024-05-26T23:04:14.324 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:14.325 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:19.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:19.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:19.327 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:19.385 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:19.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:19.438 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:19.440 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:19.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:19.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:19Z" 2024-05-26T23:04:19.443 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:19.444 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:24.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:24.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:24.446 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:24.504 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:24.505 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:24.557 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:24.558 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:24.559 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:24.560 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:24.561 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:24.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:24Z" 2024-05-26T23:04:24.563 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:24.564 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:29.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:29.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:29.566 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:29.625 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:29.625 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:29.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:29.683 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:29.684 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:29Z" 2024-05-26T23:04:29.686 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:29.687 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:34.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:34.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:34.688 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:34.801 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:34Z" 2024-05-26T23:04:34.809 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:34.811 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:39.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:39.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:39.812 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:39.871 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:39.872 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:39.924 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:39Z" 2024-05-26T23:04:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:39.930 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:44.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:44.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:44.932 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:44.992 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:44.992 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:45.045 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:45.045 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:45.045 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:45.046 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:45.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:45.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:45.049 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:45Z" 2024-05-26T23:04:45.051 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:45.052 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:50.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:50.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:50.053 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:50.110 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:50.110 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:50.160 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:50.160 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:50.160 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:50.160 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:50.161 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:50.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:50.163 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:50.164 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:50.165 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:50.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:50.165 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:50Z" 2024-05-26T23:04:50.165 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:50.166 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:04:55.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:55.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:04:55.168 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:04:55.230 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:04:55.230 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:04:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:04:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:04:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:04:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:04:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:04:55Z" 2024-05-26T23:04:55.284 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:04:55.285 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:00.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:00.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:00.286 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:00.345 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:00.345 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:00.397 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:00.397 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:00.398 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:00.399 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:00.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:00.401 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:00Z" 2024-05-26T23:05:00.402 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:00.403 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:05.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:05.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:05.405 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:05.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:05.463 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:05.515 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:05.516 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:05.517 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:05.518 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:05Z" 2024-05-26T23:05:05.519 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:05.520 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:10.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:10.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:10.521 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:10.581 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:10.581 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:10Z" 2024-05-26T23:05:10.637 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:10.638 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:15.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:15.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:15.640 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:15.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:15.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:15.751 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:15.752 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:15.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:15.754 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:15.755 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:15.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:15.755 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:15.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:15.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:15.756 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:15.758 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:15.759 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:15.759 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:15.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:15.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:15.759 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:15.760 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:15Z" 2024-05-26T23:05:15.761 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:15.762 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:20.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:20.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:20.764 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:20.824 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:20.824 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:20.883 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:20.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:20.883 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:20Z" 2024-05-26T23:05:20.883 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:20.884 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:25.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:25.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:25.885 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:25.944 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:25.996 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:25.997 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:25.998 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:25.999 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:25Z" 2024-05-26T23:05:26.000 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:26.001 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:31.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:31.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:31.003 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:31.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:31.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:31.113 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:31.114 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:31.115 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:31.116 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:31Z" 2024-05-26T23:05:31.117 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:31.118 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:36.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:36.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:36.120 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:36.180 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:36.180 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:36.231 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:36.232 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:36.233 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026361", 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.862131Z" 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:36.234 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020607", 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:52.863976Z" 2024-05-26T23:05:36.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:36Z" 2024-05-26T23:05:36.236 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:36.237 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026643 target=00000000000000000000:00000000000000026361, shard=3 master=00000000000000000000:00000000000000020889 target=00000000000000000000:00000000000000020607 2024-05-26T23:05:41.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:41.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:41.238 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:41.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.348 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:41.349 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:05:41.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:05:41.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:41.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:41Z" 2024-05-26T23:05:41.353 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:41.354 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:05:41.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.419 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:41.479 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.480 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.534 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:05:41.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.535 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: cannot find source zone id for name=* 2024-05-26T23:05:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: cannot find source zone id for name=* 2024-05-26T23:05:41.641 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-05-26T23:05:41.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.642 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.743 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:05:41.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.744 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.845 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:05:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 18, 2024-05-26T23:05:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "00ea8d6c-6a72-476f-b08d-299358d5f544", 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "test-zonegroup", 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zonegroup", 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "", 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": true, 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:05:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone3", 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8001" 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone2", 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": false, 2024-05-26T23:05:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test-zone1", 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [ 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "http://smithi007.front.sepia.ceph.com:8000" 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": false, 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": true, 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": false, 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": true, 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "", 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "supported_features": [ 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "compress-encrypted", 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [ 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-group", 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "data_flow": { 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "symmetrical": [ 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-flow1", 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "directional": [ 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "source_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "dest_zone": "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pipes": [ 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-90", 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-90", 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-91", 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-91", 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-92", 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe-xmqxus-93", 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-92", 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "7e72b8a9-5d69-4e33-98d3-490830e32745" 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "xmqxus-93", 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "5d9078f7-c6b6-41b6-8e8a-a69458435742" 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "sync-pipe", 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "*" 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": { 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket": "*", 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "*" 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "params": { 2024-05-26T23:05:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "source": { 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "filter": { 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [] 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "dest": {}, 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "priority": 0, 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "system", 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "user": "" 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "allowed" 2024-05-26T23:05:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled_features": [ 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "resharding" 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "2be55b7a-f70c-4aa7-a1f4-aea3d2de43be", 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3864594665 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "5d9078f7-c6b6-41b6-8e8a-a69458435742", 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3171627794 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 3772860827 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "test-zonegroup", 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "7e72b8a9-5d69-4e33-98d3-490830e32745", 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-05-26T23:05:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:05:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:05:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:05:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:05:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:05:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-05-26T23:05:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-05-26T23:05:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-05-26T23:05:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "a03a6f99-1643-481f-8ae9-2762eec2e396", 2024-05-26T23:05:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "test-realm", 2024-05-26T23:05:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:05:41.872 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:05:41.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.873 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:41.922 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:41.922 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:41.969 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:05:41.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:41.970 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xmqxus-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.136 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-05-26T23:05:42.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.137 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket xmqxus-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.332 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-05-26T23:05:42.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xmqxus-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=xmqxus-92', '--source-bucket=*', '--dest-bucket=xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xmqxus-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=xmqxus-92', '--source-bucket=*', '--dest-bucket=xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.333 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-xmqxus-92 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=xmqxus-92 '--source-bucket=*' --dest-bucket=xmqxus-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.415 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.504 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-xmqxus-92","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-05-26T23:05:42.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xmqxus-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=xmqxus-92', '--source-bucket=*', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xmqxus-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=xmqxus-92', '--source-bucket=*', '--dest-bucket=xmqxus-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.504 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-xmqxus-93 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=xmqxus-92 '--source-bucket=*' --dest-bucket=xmqxus-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-xmqxus-92","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-05-26T23:05:42.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xmqxus-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.674 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xmqxus-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.780 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-xmqxus-92","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-05-26T23:05:42.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-05-26T23:05:42.781 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-05-26T23:05:42.826 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.826 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.874 INFO:teuthology.orchestra.run.smithi007.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-xmqxus-90","source":{"bucket":"xmqxus-90","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-91","source":{"bucket":"xmqxus-91","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-92","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xmqxus-93","source":{"bucket":"xmqxus-92","zones":["test-zone1"]},"dest":{"bucket":"xmqxus-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-05-26T23:05:42.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:42.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:42.874 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:42.955 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:05:42.969 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:05:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:05:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:05:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:05:42.765896Z" 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:05:42.970 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:05:42.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:05:42.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:42.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:42.972 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:43.023 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:43.024 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:43.054 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:43.054+0000 7f87f4144a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:43.055 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:43.055+0000 7f87f4144a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:43.069 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:43.069 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:43.069 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:43.070 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:05:43.071 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:43.071 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:05:43.073 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:43.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:43.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:43.073 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:43.074 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:43.075 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:05:43.076 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:05:43.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:43.077 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:43Z" 2024-05-26T23:05:43.077 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:43.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:05:48.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:48.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:48.079 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:48.136 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:48.137 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:48.171 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:48.172+0000 7f09ca70ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:48.172 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:48.173+0000 7f09ca70ea80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:05:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:05:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:48Z" 2024-05-26T23:05:48.193 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:48.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:05:53.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:53.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:53.196 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:53.252 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:53.252 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:53.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:53.289+0000 7f618cfada80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:53.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:53.290+0000 7f618cfada80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:53.304 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:53.304 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:53.304 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:53.304 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:53.304 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:05:53.306 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:53.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:53.308 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:53Z" 2024-05-26T23:05:53.309 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:53.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:05:58.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:58.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:05:58.312 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:05:58.368 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:05:58.368 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:05:58.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:58.404+0000 7f99dab78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:58.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:05:58.405+0000 7f99dab78a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:58.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:58.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:05:58.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:05:58Z" 2024-05-26T23:05:58.424 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:05:58.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:06:03.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:03.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:03.427 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:03.483 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:03.483 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:03.517 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:03.517+0000 7f65ef8dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:03.517 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:03.518+0000 7f65ef8dfa80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:03.532 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:03.533 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:03.534 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:03.536 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:03Z" 2024-05-26T23:06:03.537 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:03.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:06:08.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:08.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:08.539 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:08.597 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:08.597 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:08.631 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:08.631+0000 7fe2bf8d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:08.632 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:08.632+0000 7fe2bf8d6a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:08.646 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:08.646 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:08.647 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:08.648 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764616.393027_1523.1", 2024-05-26T23:06:08.649 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:03:36.393027Z", 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:08.650 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:08Z" 2024-05-26T23:06:08.651 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:08.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764616.393027_1523.1 2024-05-26T23:06:13.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:13.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:13.653 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:13.711 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:13.711 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:13.744 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:13.744+0000 7f26c0cdba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:13.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:13.745+0000 7f26c0cdba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:13.760 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:13.760 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:13.760 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:13.761 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:13.762 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764742.098674_1536.1", 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:13.763 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:05:42.098674Z", 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:13Z" 2024-05-26T23:06:13.765 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:13.767 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1716764742.765896_1556.1 target=1_1716764742.098674_1536.1 2024-05-26T23:06:18.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:18.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:18.768 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:18.826 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:18.827 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:18.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:18.860+0000 7fc8d58aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:18.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:18.861+0000 7fc8d58aba80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:18.876 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:18.877 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:18.878 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:05:42.765896Z", 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:18.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:18.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:18Z" 2024-05-26T23:06:18.881 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:18.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:06:18.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:06:18.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:18.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:18.883 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:18.941 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:18.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:06:18.977+0000 7f6ece7cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:18.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:06:18.978+0000 7f6ece7cca80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:18.993 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:18.994 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:18.995 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:05:42.765896Z", 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:18.997 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:06:18Z" 2024-05-26T23:06:18.998 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:06:18.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:06:19.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.028 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:No --period given, using current period=de3b77f2-60b7-47e4-b23e-f2e513d5e591 2024-05-26T23:06:19.126 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:06:19.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.883216Z" 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:06.899912Z" 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:05:42.765896Z" 2024-05-26T23:06:19.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.129 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:19.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:15:07.923166Z" 2024-05-26T23:06:19.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.129 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:06:19.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-05-26T23:06:19.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.131 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:19.183 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:19.183 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:19.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:19.215+0000 7f626c963a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:19.216 INFO:teuthology.orchestra.run.smithi137.stderr:2024-05-26T23:06:19.216+0000 7f626c963a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:19.230 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:19.230 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:19.230 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.231 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.232 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:05:42.765896Z", 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:19.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:19Z" 2024-05-26T23:06:19.235 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:19.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-05-26T23:06:19.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-05-26T23:06:19.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.237 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:19.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:06:19.319+0000 7f2bbb728a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:19.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-05-26T23:06:19.320+0000 7f2bbb728a80 20 RGW-SYNC:meta: read sync status 2024-05-26T23:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-05-26T23:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": { 2024-05-26T23:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "info": { 2024-05-26T23:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "sync", 2024-05-26T23:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "num_shards": 4, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "period": "de3b77f2-60b7-47e4-b23e-f2e513d5e591", 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "markers": [ 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 0, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.883216_190.1", 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 1, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.883216Z", 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 1, 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761706.899912_718.1", 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:06.899912Z", 2024-05-26T23:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 2, 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716764742.765896_1556.1", 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T23:05:42.765896Z", 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "key": 3, 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "state": 1, 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "1_1716761707.923166_650.1", 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "next_step_marker": "", 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "total_entries": 0, 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pos": 0, 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "timestamp": "2024-05-26T22:15:07.923166Z", 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "full_sync": { 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "total": 1, 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "complete": 1 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "current_time": "2024-05-26T23:06:19Z" 2024-05-26T23:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-05-26T23:06:19.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-05-26T23:06:19.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.340 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:19.446 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000014646", 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:06:19.028268Z" 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000026784", 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:06:19.017690Z" 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:19.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-05-26T23:06:19.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-05-26T23:06:19.448 INFO:teuthology.orchestra.run.smithi007.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:19.448 INFO:teuthology.orchestra.run.smithi007.stdout: "last_update": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:19.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-05-26T23:06:19.448 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-05-26T23:06:19.449 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:06:19.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:19.449 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:19.502 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:19.502 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:19.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.549 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:19.550 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:19.551 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:19Z" 2024-05-26T23:06:19.552 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:19.553 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:24.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:24.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:24.554 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:24.612 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:24.613 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:24.664 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:24.664 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:24.664 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:24.665 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:24.666 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:24.667 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:24.668 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:24Z" 2024-05-26T23:06:24.670 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:24.671 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:29.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:29.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:29.672 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:29.731 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:29.731 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:29.783 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:29.784 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:29.785 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:29.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:29.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:29.789 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:29Z" 2024-05-26T23:06:29.789 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:29.790 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:34.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:34.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:34.792 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:34.850 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:34.851 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:34Z" 2024-05-26T23:06:34.909 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:34.910 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:39.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:39.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:39.911 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:39.970 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:39.970 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:40.022 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:40.023 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:40.027 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:40.027 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:40.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:40.027 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:40Z" 2024-05-26T23:06:40.027 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:40.028 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:45.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:45.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:45.030 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:45.088 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:45.089 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:45.140 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:45.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:45.143 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:45.144 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:45.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:45.145 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:45Z" 2024-05-26T23:06:45.145 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:45.146 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:50.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:50.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:50.147 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:50.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:50.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:50.258 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:50.258 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:50.258 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:50.258 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:50.259 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:50.260 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:50.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:50.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:50Z" 2024-05-26T23:06:50.263 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:50.264 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:06:55.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:55.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:06:55.266 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:06:55.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:06:55.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:06:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:06:55.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.381 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:06:55.382 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:06:55.382 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:06:55.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:06:55.382 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:06:55Z" 2024-05-26T23:06:55.382 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:06:55.383 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:00.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:00.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:00.384 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:00.440 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:00.440 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:00.491 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:00.491 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:00.491 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:00.491 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:00.492 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:00.493 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:00.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:00Z" 2024-05-26T23:07:00.496 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:00.497 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:05.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:05.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:05.499 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:05.558 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:05.558 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:05.609 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:05.609 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:05.609 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:05.610 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:05.611 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:05.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:05.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:05Z" 2024-05-26T23:07:05.614 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:05.615 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:10.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:10.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:10.616 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:10.676 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:10.676 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:10.728 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:10Z" 2024-05-26T23:07:10.736 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:10.737 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:15.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:15.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:15.739 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:15.798 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:15.798 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:15.849 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:15Z" 2024-05-26T23:07:15.854 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:15.855 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:20.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:20.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:20.857 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:20.915 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:20.915 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:20.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:20.970 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:20.971 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:20.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:20.973 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:20Z" 2024-05-26T23:07:20.973 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:20.973 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:25.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:25.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:25.975 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:26.088 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:26.088 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:26.088 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:26.088 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:26.088 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:26.089 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:26.091 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.092 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:26.093 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:26.093 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:26.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:26.093 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:26Z" 2024-05-26T23:07:26.093 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:26.094 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:31.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:31.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:31.096 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:31.156 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:31.156 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:31.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:31Z" 2024-05-26T23:07:31.212 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:31.213 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:36.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:36.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:36.214 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:36.275 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:36.275 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.334 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:36.337 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:36.338 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:36Z" 2024-05-26T23:07:36.339 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:36.340 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:41.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:41.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:41.341 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:41.400 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:41.400 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:41.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014505", 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:02:03.058723Z" 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:41.452 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026643", 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.352949Z" 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:41.453 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:41.454 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:41Z" 2024-05-26T23:07:41.455 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:41.456 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000014646 target=00000000000000000000:00000000000000014505, shard=1 master=00000000000000000000:00000000000000026784 target=00000000000000000000:00000000000000026643 2024-05-26T23:07:46.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:46.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-05-26T23:07:46.458 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-05-26T23:07:46.518 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-05-26T23:07:46.518 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "instance_id": 15937190248483520075 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:46.570 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000014646", 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 9, 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:06:19.028268Z" 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:46.571 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000026784", 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 69, 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:06:19.017690Z" 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000022966", 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T22:53:22.056415Z" 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "incremental-sync", 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "00000000000000000000:00000000000000020889", 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-05-26T23:07:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-05-26T23:04:08.391163Z" 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 78, 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 78 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-05-26T23:07:46Z" 2024-05-26T23:07:46.574 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-05-26T23:07:46.575 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-05-26T23:07:46.636 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.637 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.638 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 933, in test_multi_period_incremental_sync 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.640 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.642 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1061, in test_multi_zone_redirect 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.643 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1090, in test_zonegroup_remove 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.644 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1173, in test_bucket_sync_disable 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.645 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1197, in test_bucket_sync_enable_right_after_disable 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.646 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1232, in test_bucket_sync_disable_enable 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.647 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1276, in test_multipart_object_sync 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.648 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1313, in test_encrypted_object_sync 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.649 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1343, in test_bucket_index_log_trim 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-05-26T23:07:46.650 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1338, in make_test_bucket 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1397, in test_bucket_reshard_index_log_trim 2024-05-26T23:07:46.651 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1393, in make_test_bucket 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.652 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1480, in test_bucket_reshard_incremental 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.653 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1531, in test_bucket_reshard_full 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.654 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.655 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1624, in test_bucket_sync_run_basic_incremental 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1691, in test_zap_init_bucket_sync_run 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-05-26T23:07:46.657 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1769, in test_bucket_full_sync_after_data_sync_init 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.658 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.659 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1827, in test_bucket_incremental_sync_after_data_sync_init 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.660 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.661 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 1915, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-05-26T23:07:46.662 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests:AssertionError 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2340, in test_sync_flow_directional_zonegroup_select 2024-05-26T23:07:46.663 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesA, content) 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2622, in test_sync_different_buckets 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesC, content) 2024-05-26T23:07:46.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2725, in test_sync_multiple_buckets_to_single 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: check_objects_not_exist(bucket, objnamesA) 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2088, in check_objects_not_exist 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: check_object_not_exists(bucket, objname) 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2084, in check_object_not_exists 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: assert_equal(k, None) 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests:AssertionError: != None 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-05-26T23:07:46.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2890, in test_sync_single_bucket_to_multiple 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 9905.461s 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests: 2024-05-26T23:07:46.667 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, failures=26) 2024-05-26T23:07:46.668 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-05-26T23:07:46.908 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=07b268da26434c0da97905240f6ae7b4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_f1882d8093fb4b60c1d1e52444565787ca4bd6cc/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-05-26T23:07:46.909 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-05-26T23:07:46.922 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-05-26T23:07:46.979 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-05-26T23:07:47.007 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-05-26T23:07:47.008 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:07:47.011 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T23:07:47.008+0000 7f30b39ef700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 49399) UID: 0 2024-05-26T23:07:47.024 INFO:tasks.rgw.c1.client.0.smithi007.stdout:2024-05-26T23:07:47.008+0000 7f30b7c5ca80 -1 shutting down 2024-05-26T23:07:48.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-05-26T23:07:53.110 INFO:tasks.rgw.c1.client.0:Stopped 2024-05-26T23:07:53.110 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-05-26T23:07:53.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/vault-root-token 2024-05-26T23:07:53.208 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-05-26T23:07:53.208 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:07:53.249 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T23:07:53.249+0000 7f27263cf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 146759) UID: 0 2024-05-26T23:07:53.250 INFO:tasks.rgw.c1.client.1.smithi007.stdout:2024-05-26T23:07:53.249+0000 7f272a63ca80 -1 shutting down 2024-05-26T23:07:54.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-05-26T23:07:54.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-05-26T23:07:59.310 INFO:tasks.rgw.c1.client.1:Stopped 2024-05-26T23:07:59.311 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-05-26T23:07:59.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/vault-root-token 2024-05-26T23:07:59.407 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-05-26T23:07:59.407 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:07:59.408 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T23:07:59.408+0000 7fce7aec5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 72619) UID: 0 2024-05-26T23:07:59.408 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-05-26T23:07:59.408+0000 7fce7f132a80 -1 shutting down 2024-05-26T23:07:59.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-05-26T23:08:04.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-05-26T23:08:05.509 INFO:tasks.rgw.c2.client.0:Stopped 2024-05-26T23:08:05.510 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-05-26T23:08:05.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/vault-root-token 2024-05-26T23:08:05.604 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-05-26T23:08:05.604 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:05.645 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-05-26T23:08:05.645+0000 7faf017a3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 47361) UID: 0 2024-05-26T23:08:05.646 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-05-26T23:08:05.645+0000 7faf05a10a80 -1 shutting down 2024-05-26T23:08:10.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-05-26T23:08:10.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-05-26T23:08:11.706 INFO:tasks.rgw.c2.client.1:Stopped 2024-05-26T23:08:11.707 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-05-26T23:08:11.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/vault-root-token 2024-05-26T23:08:11.801 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-26T23:08:11.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-05-26T23:08:12.191 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-26T23:08:12.191 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-26T23:08:12.192 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-05-26T23:08:12.192 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:12.234 INFO:tasks.ceph.c2.osd.0:Stopped 2024-05-26T23:08:12.234 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-05-26T23:08:12.235 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:12.271 INFO:tasks.ceph.c2.osd.1:Stopped 2024-05-26T23:08:12.272 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-05-26T23:08:12.272 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:12.343 INFO:tasks.ceph.c2.osd.2:Stopped 2024-05-26T23:08:12.343 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-26T23:08:12.344 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-05-26T23:08:12.344 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:12.408 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-05-26T23:08:12.408 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-26T23:08:12.409 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-05-26T23:08:12.409 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:12.468 INFO:tasks.ceph.c2.mon.a:Stopped 2024-05-26T23:08:12.468 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-26T23:08:12.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-05-26T23:08:12.537 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T23:08:12.537 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-05-26T23:08:12.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T23:08:12.851 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-05-26T23:08:12.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T23:08:12.944 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-05-26T23:08:13.022 INFO:tasks.ceph:Archiving mon data... 2024-05-26T23:08:13.024 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/data/c2.mon.a.tgz 2024-05-26T23:08:13.024 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-05-26T23:08:13.041 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.wisyyQpM4w 2024-05-26T23:08:13.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.wisyyQpM4w 2024-05-26T23:08:13.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.wisyyQpM4w 2024-05-26T23:08:13.277 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.wisyyQpM4w is 370KB 2024-05-26T23:08:13.374 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.wisyyQpM4w 2024-05-26T23:08:13.391 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-26T23:08:13.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-26T23:08:13.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-26T23:08:13.458 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-26T23:08:13.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-05-26T23:08:13.995 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-26T23:08:13.995 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-26T23:08:13.995 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-05-26T23:08:13.995 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:14.077 INFO:tasks.ceph.c1.osd.0:Stopped 2024-05-26T23:08:14.078 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-05-26T23:08:14.078 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:14.140 INFO:tasks.ceph.c1.osd.1:Stopped 2024-05-26T23:08:14.141 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-05-26T23:08:14.141 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:14.206 INFO:tasks.ceph.c1.osd.2:Stopped 2024-05-26T23:08:14.206 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-26T23:08:14.206 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-05-26T23:08:14.206 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:14.231 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-05-26T23:08:14.231 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-26T23:08:14.231 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-05-26T23:08:14.231 INFO:teuthology.orchestra.run:waiting for 300 2024-05-26T23:08:14.297 INFO:tasks.ceph.c1.mon.a:Stopped 2024-05-26T23:08:14.298 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-26T23:08:14.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-05-26T23:08:14.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T23:08:14.431 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-05-26T23:08:14.794 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T23:08:14.795 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-05-26T23:08:14.899 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T23:08:14.899 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-05-26T23:08:14.997 INFO:tasks.ceph:Archiving mon data... 2024-05-26T23:08:14.998 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/data/c1.mon.a.tgz 2024-05-26T23:08:14.998 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-05-26T23:08:15.017 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.ZHf2fvQ9VA 2024-05-26T23:08:15.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.ZHf2fvQ9VA 2024-05-26T23:08:15.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.ZHf2fvQ9VA 2024-05-26T23:08:15.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-05-26T23:08:15.313 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.ZHf2fvQ9VA is 426KB 2024-05-26T23:08:15.411 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.ZHf2fvQ9VA 2024-05-26T23:08:15.428 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-26T23:08:15.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-26T23:08:15.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-26T23:08:15.502 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-05-26T23:08:15.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-05-26T23:08:15.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-05-26T23:08:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-05-26T23:08:15.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-05-26T23:08:15.582 INFO:tasks.ceph:Archiving crash dumps... 2024-05-26T23:08:15.584 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi007/crash 2024-05-26T23:08:15.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-26T23:08:15.637 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi137/crash 2024-05-26T23:08:15.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-26T23:08:15.667 INFO:tasks.ceph:Compressing logs... 2024-05-26T23:08:15.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T23:08:15.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T23:20:20.500 INFO:tasks.ceph:Archiving logs... 2024-05-26T23:20:20.501 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi007/log 2024-05-26T23:20:20.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-26T23:21:38.989 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi137/log 2024-05-26T23:21:38.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-26T23:21:53.234 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-26T23:21:53.244 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-26T23:21:53.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-26T23:21:53.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-26T23:21:53.376 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-26T23:21:53.385 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-26T23:21:53.385 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T23:21:53.388 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T23:21:53.405 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-05-26T23:21:53.406 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2024-05-26T23:21:53.409 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T23:21:53.409 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-05-26T23:21:53.409 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 7 377 100 +686us[ +596us] +/- 55ms 2024-05-26T23:21:53.410 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -718us[ -718us] +/- 53ms 2024-05-26T23:21:53.410 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +45us[ +45us] +/- 91ms 2024-05-26T23:21:53.410 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +636us[ +608us] +/- 56ms 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 -770us[ -770us] +/- 53ms 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +20us[ +20us] +/- 91ms 2024-05-26T23:21:53.424 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T23:21:53.425 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-26T23:21:53.435 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-26T23:21:53.435 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-26T23:21:53.454 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-26T23:21:53.574 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-26T23:21:53.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-26T23:21:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T23:21:53.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-26T23:21:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T23:21:53.703 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-26T23:21:53.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-26T23:21:53.737 INFO:teuthology.task.internal:Duration was 11599.633258 seconds 2024-05-26T23:21:53.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-26T23:21:53.768 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-26T23:21:53.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-26T23:21:53.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-26T23:21:53.817 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T23:21:53.864 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T23:21:54.133 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-26T23:21:54.133 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-05-26T23:21:54.134 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-26T23:21:54.161 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-05-26T23:21:54.162 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-26T23:21:54.184 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-26T23:21:54.184 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T23:21:54.204 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T23:21:54.267 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-26T23:21:54.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-26T23:21:54.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-26T23:21:54.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-26T23:21:54.387 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-26T23:21:54.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-26T23:21:54.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-26T23:21:54.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-26T23:21:54.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-26T23:21:54.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-26T23:21:54.512 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-05-26T23:21:54.525 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-05-26T23:21:54.547 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-26T23:21:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T23:21:54.606 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-26T23:21:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T23:21:54.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-26T23:21:54.633 INFO:teuthology.task.internal:Transferring archived files... 2024-05-26T23:21:54.634 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi007 2024-05-26T23:21:54.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-26T23:21:54.687 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-24_22:40:03-rgw-reef-distro-default-smithi/7725509/remote/smithi137 2024-05-26T23:21:54.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-26T23:21:54.727 INFO:teuthology.task.internal:Removing archive directory... 2024-05-26T23:21:54.727 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-26T23:21:54.729 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-26T23:21:54.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-26T23:21:54.791 INFO:teuthology.task.internal:Not uploading archives. 2024-05-26T23:21:54.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-26T23:21:54.813 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-26T23:21:54.813 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-26T23:21:54.815 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-26T23:21:54.832 INFO:teuthology.orchestra.run.smithi007.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 26 23:21 /home/ubuntu/cephtest 2024-05-26T23:21:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 26 23:21 /home/ubuntu/cephtest 2024-05-26T23:21:54.838 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-26T23:21:54.848 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-26T23:21:54.916 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} duration: 11599.633257865906 failure_reason: rgw multisite test failures flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=07b268da26434c0da97905240f6ae7b4 status: fail success: false 2024-05-26T23:21:54.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-26T23:21:54.998 INFO:teuthology.run:FAIL